August 1, 2017
Today, Brad and Pippin will be talking about what they’ve been up to the past few months, including things that have been going well and things that they’ve struggled with.
Some of the highlights of the show include:
- The challenges that EDD faced when developing Commissions beta 3.4, as well as why the release was planned and developed.
- What EDD is planning to release over the next year or two.
- Brad’s newest blog post about creating your own certificate authority.
- Details about Mergebot’s launch.
- How Serialized Editor has helped Brad.
- Details of the 2.8 release of EDD’s core plugin.
- The status of the Migrate DB Pro release, including details of the new import feature.
- The importance of not working in a vacuum.
- The features of Restrict Content Pro 2.9.
Links and Resources:
If you’re enjoying the show we sure would appreciate a Review in iTunes. Thanks!
BRAD: Welcome to Episode 82. This time Pippin and I will be talking about what we’ve been up to in the past couple months, some things that worked out for us, some things that we struggled with. Pippin, what have you been up to, man?
PIPPIN: Well, it’s been, I think, almost a month and a half to two months since our last episode when we interviewed Matt Mullenweg, so quite a bit has actually happened in that month to month and a half. I kind of forget how fast things move until you look back and think, that wasn’t very long ago, and then you look at all the different things that happened in that time.
We’ve had three major updates that we pushed out in the last couple of days. Actually, all of this week, I think. Two of them went live yesterday and one the day before that were all the culmination of anywhere from one to six months of work.
All right, let me give you a quick back story. Five years ago, we started Easy Digital Downloads. And, like a lot of people in the WordPress world, we were very excited about what custom post types allowed us to do. There was this general philosophy that proliferated the WordPress development world that you should use custom post types for anything and everything, and so we used custom post types and post meta to store transactional data for Easy Digital Downloads: payment records, logs, et cetera. Well, that was five years ago.
As a lot of people who have used a WordPress e-commerce plugin, whether you’re talking EDD or WooCommerce or any of the others that also use custom post types or any similar scenario, we’ve discovered that there are significant problems and side effects posed by putting that data in that database. It basically comes down to you’re storing a whole lot of data in a super inefficient manner in a way that it was never designed to be used for. You’re putting a square into a round hole is what you’re trying to do, basically.
The reason we did it, the same reason everybody else did it, is that it was super, super easy to do. There was almost zero learning curve to do it, and we could do it. Anyway, that was five years ago.
We’ve had serious — there’s been a lot of serious problems caused by it, especially with performance and scaling. Well, for the last two years, we’ve been planning how to get that data out, how to move it into custom database tables that are precisely designed for the schema that we need, precisely designed for the purpose that the data has, what it is, where it needs to be stored, what we need to store, how we need to store it, what kind of relationships we need, et cetera. We’ve been working on slowly planning this out for two years, and yesterday we shipped the first one, the first migration.
We have a plugin called Commissions that allows store owners to track commission earnings for vendors on their website. If you have a website that has products from multiple vendors, you can track earnings through this Commissions plugin. All of those records were stored as a post type, just like a lot of other data inside of EDD and related plugins. We shipped our first, the first beta of version 3.4 yesterday, and it includes a migration into a custom table designed specifically for the purpose of commission tracking.
We first started testing this about three weeks ago on our own sites. We first did it on a very small site that had a little bit of data, a few thousand records at most. We tested the migration, and then we tested also just the general functionality of the plugin. Like after the data is migrated, does everything still work? We tested the migration. Then we also have been working on building a backwards compatibility layer.
Then, when all that was done and everything was working on that small site, we put it onto our big site, EasyDigitalDownloads.com. Our main store site has been running this custom database table for commission records for two weeks now. Anyway, so far it’s been a super successful project.
BRAD: Cool. Yeah. Did you guys find it really challenging to do this release, to make sure that everything is backwards compatible, or was it not that bad?
PIPPIN: Well, okay, so yes and no. I mean I think there’s a couple of major challenges. Your first major challenge is how do you successfully get all the data migrated safely, not only to make sure that you have data integrity, but also make sure, like, is this going to work on low end hosting accounts? That was one challenge.
That was definitely difficult. However, it’s something that we’ve been doing a lot of in the last three years. We’ve got pretty solid batch processing and upgrade scripts that we can use now that take care of that.
Okay, so first of all, let me tell you what we did for backwards compatibility. We built a complete layer, so all of the data used to live in the WP post and the WP post meta. We tried to look at all of the data that other plugins or people/developers might be querying and they might be using. We wanted to make sure that those queries still worked even when the data did not exist inside of the WordPress post table any more.
For example, if you call get_post_meta on a meta key that was associated with an EDD commission post entry and that data no longer exists, your query will still work. We wanted to build a backwards compatibility layer that made sure that nobody’s data became inaccessible even if they were using the old API. That was challenging mostly in trying to predict what people are doing. The actual compatibility layer is not that challenging. That’s pretty simple, honestly. All you’re doing is rerouting queries.
The problem is that you really do have to try to predict what cases you need to account for. You can say, well, we have these known meta keys. We know where that data used to live, and we know where the data lives now. We can do that.
But what if somebody has registered their own custom meta key and we don’t have any way to know about that? We have to then think, okay, maybe we can provide a way for developers to register a compatibility key, so like to say, “Hey, I have an old meta key. I want you to reroute my calls for me.” Okay. We can do that. We built that.
What about when somebody tries to update an old meta value that no longer exists? What happens? Well, it just works. That was challenging making sure that all of that kind of stuff happens.
BRAD: That’s a compatibility layer that you’ve added to the get_post_meta, update_post_meta, delete_post_meta, those. There’s hooks inside those WordPress functions, and so you’ve hooked into those, and you’re looking for specific keys. And I guess you’re probably looking up the post that’s being referenced.
BRAD: And to check that it’s an EDD post type, I guess.
PIPPIN: But keep in mind that that data doesn’t exist any more because, once it’s migrated out, it’s deleted.
BRAD: You’re not keeping a record of that ID, like what it used to be?
PIPPIN: We keep a legacy ID.
PIPPIN: We have that stored. That is there.
PIPPIN: But it doesn’t belong in WP post any more.
BRAD: Right. But is that what you do?
BRAD: …compatibility layer.
PIPPIN: Yeah, we basically look it up to say, hey, is this a legacy ID?
PIPPIN: Okay. We need to move it over.
BRAD: Right. Right, and because you have that in a custom table in its own column, I’m assuming, then it’s a nice, quick lookup, I’m guessing.
PIPPIN: Yeah, it’s pretty easy to look it up.
BRAD: Yeah. Okay, that’s pretty cool.
PIPPIN: So far it’s been pretty successful. For our test site, we migrated 36,000 records. That migration took about 30 minutes to do, maybe 45. It was kind of cool.
We looked at the stats and, when we finished the migration, our amounts were off by like one penny. Sorry. They were off by like $10. We were wondering, where did the $10 come from? Well, what we realized is that we were doing this on a live site, and we had a sale come through during our migration. And so the existing — even as we were migrating that data, it still worked. It put that new sale and got recorded in the new table while we were inserting all of this old back history into it.
At first it freaked us out because we were like, where did it come from? Like, why do we have extra? It turns out it worked perfectly, and we had everything down to the penny, down to the exact number of records, et cetera.
This is one piece of a much larger project. Our project for 2017 was to get all of our data out of custom tables, and we basically have four sets of data–
BRAD: Out of the post tables, you mean.
PIPPIN: Out of the post tables.
PIPPIN: Yeah, sorry.
BRAD: Into custom tables.
PIPPIN: Into custom tables.
PIPPIN: There is basically four major sets of data. There’s commission records. There’s license keys. There’s log entries. And there’s payment records. This is our first one to do because it’s the smallest, and so it was — if something is going to go wrong, we’d rather it happen here.
This being the first one, we’re then going to be able to do our next ones. We’ll do license keys. We’ll do payment records. And we’ll do logs. All of those will happen during this year still.
Well, because of the old database schema where you’re using WP post and WP post meta, and those tables are not designed for this kind of data, what it really means is that you have really severe bloating of your database. As an example, “one” entry in the database, in the old schema, would actually take up 11 rows in your database, or maybe 20 rows, or maybe 5 rows because of all the meta data stored as separate rows that really should belong in its own column.
When we ran this migration, we moved 36,000 records. Doing that allowed us to delete over 250,000 rows from the database. That’s the smallest of our four data sets. So when we do our next ones, we’re going to delete another 300,000 to 500,000 rows from our database. This is just our database. This is not counting everybody else’s who have much bigger sites than us. Then when we do the next one, we’re going to delete a million rows from the database.
PIPPIN: It’s crazy.
BRAD: What is your strategy, or maybe you haven’t even thought about this yet–I don’t know–about taking out the backwards compatibility layer. Is that ever going to come out, or are you just going to leave it in there?
PIPPIN: Yeah, it’ll probably come out at some point. Or, at a minimum, we’ll put it in there to throw hard notices to get it to see when it’s used to make sure that–
BRAD: Does it throw out deprecated notices for now?
PIPPIN: Right now, if you have debug on and you’re signed in as an admin, you will get a “doing it wrong” notice. But it only happens if you’re logged in as an admin and debug is enabled because we don’t want to cause any unexpected behavior for customers. But, yeah, so I think eventually we will absolutely hard deprecate them to the point where they either — maybe they don’t work and they always throw a notice because, if we migrated the data and you still have the old methods in place a year and a half later, it’s time to get those fixed. I don’t know if it’s a year. I don’t know if it’s two years. But, yes, it will happen.
BRAD: Yeah, eventually. Clean up time.
BRAD: Yeah. The migration routine looks like — does that just reload the page, or is it Ajax?
PIPPIN: It is. There are two ways to run this. Our standard upgrade routine inside of EDD uses a combination of Ajax and page reloads to do a batch processing that can, even on slow servers, go through tens of thousands of steps.
BRAD: Oh, is it the same library that Drew was on this very program talking about?
PIPPIN: Very similar. When Drew came on a few months ago to talk about batch processing in Affiliate WP, the batch processor that he built was used to some of the EDD batch processing as a starting point.
PIPPIN: But it’s not the same one, but it is similar.
PIPPIN: But we also added the option to run the migration via WPCLI, which is definitely the recommended way.
BRAD: Yeah. I love that. That’s a smart move because, yeah, if you have a giant site, you don’t want to have to do it via the UI.
PIPPIN: No. No.
BRAD: You’re just asking for trouble.
PIPPIN: Yeah, like our site, we moved 36,000 commission records, removed 230,000 records from the database. That took us about 30 to 45 minutes via WPCLI. If we had done that in the UI, it probably would have been three hours, easily.
BRAD: You guys didn’t take your site down temporarily–
BRAD: –to run this?
PIPPIN: No, we ran it live.
BRAD: Oh, man.
PIPPIN: We wanted to make sure that it worked live.
BRAD: You guys — yeah, you guys are gutsy, man.
PIPPIN: We tested it on our staging site first.
BRAD: This is a big, big step for you guys. I think this is going to be a huge advantage over the other e-commerce plugins that are out there because, I mean, are there any other e-commerce plugins that actually use custom tables?
PIPPIN: WP eCommerce does.
BRAD: It does?
PIPPIN: They’ve used custom tables since day one.
BRAD: Okay. Yeah, and it was really controversial, wasn’t it?
PIPPIN: It was. Right. They are actually, I believe — Justin, if you’re listening, I’m sorry if I’m wrong on this, but I believe WP eCommerce was actually one of the reasons that a lot of people preached for using custom tables for all of this kind of data because they’re like, “Why would you use custom tables? That’s wrong. Just use the natural WordPress APIs. See, don’t do what WP eCommerce is doing.” Now we realized X years later, wow, what a horrible mistake this was. WP eCommerce, you were right all along.
BRAD: Yeah. That’s a long time to wait for vindication, though.
PIPPIN: Yeah. Absolutely. Yeah, so we’re really happy to have this update out. If you’re interested in reading about it, go to EasyDigitalDownloads.com/development, and it’s the EDD Commissions 3.4 release. We’re going to be doing four more of these, three or four more of these in the next six months if you’re interested. By the end of 2017, we want to have all of our data out of custom post types with the one exception of we’re not going to move products out because a product makes sense as a post.
Anyway, that’s been the first of our major projects. Brad, what’s been one of your recent projects in the last month and a half since we talked last?
BRAD: I just wrote an article for our blog, and I had to learn some stuff. It was about creating your own certificate authority, which I didn’t even know about. I don’t know, a couple months ago I wrote a post about how to create your own self-signed certificates and install them into your local Web server for local development so that you can work with HTTPS locally.
One of the comments, they said, oh, you could also set up your own certificate authority so that you didn’t have to add the certificate to Mac OS X Keychain or any other of your devices so that you only have to add your certificate, your certificate authority certificate that one time, and then any certificates that you sign with that CA will just work. They’ll just be accepted.
I was like, oh, man. That sounds so much better than having to add — every time you add a new site to have to create a certificate, but then also add it to your Keychain or any other devices. It’s just a huge pain in the butt. You just do it once when you have a certificate authority.
That turned out to be really easy, actually. It’s super easy to create your own certificate authority. It sounds like it’s really hard, right? Because Verisign, you know, has been the certificate authority for ages. We only recently got the free one, Let’s Encrypt. Amazon is now a certificate authority. Just fairly recently they became one.
PIPPIN: Right. I can’t compete with Amazon. How can I be a certificate authority?
BRAD: Yeah, exactly. Well, the kind of mechanics of it are super simple, though. It’s just like you run a few commands in the command line and it outputs a few files. Voila, you are now a certificate authority.
The trick, though, is to get that certificate on every machine in the world. That’s the hard part because Verisign, Let’s Encrypt, their certificates are bundled with Mac OS X or Firefox or whatever. They’re included in the software that needs them, that needs to recognize them as certificate authorities. Yeah, you become a certificate authority, but not really, right? You have the ingredients, but not — you still need to get it on every system in the world, right?
Have you ever looked into that kind of stuff?
PIPPIN: I’ve never set up assign authority for myself. I’ve done self-signed certificates, but no. I kind of want to try this now, so I’m going to put it on my list for later this week. I’m going to make it work on my local sites, and I’ll let you know if it works.
You guys are really excellent about taking, well, let’s just say a complex topic of something and doing super in depth posts on it. For anybody that’s listening, if you guys don’t read and don’t subscribe to the Delicious Brains blog, you should because it’s really, really excellent. I really like what you do with — you said it when you first started talking about creating your own certificate authorities saying, “I don’t know how this works. I don’t know how to do it, so I did some research so I could write about it.” I love that.
BRAD: Yeah. Thanks. I mean that means a lot coming from you because, I mean, you’re like one of the biggest education guys in WordPress.
PIPPIN: Well, maybe used to be. I don’t write much any more.
BRAD: Yeah. Well, yeah, that’s right. PippinsPlugins.com, though, I think there’s probably a lot of developers that probably started their education on your articles, so yeah, it’s good stuff.
What else have we been up to? Mergebot should be launched now. By the time that this actually gets published, Mergebot, we should have announced it. Mergebot is kind of, you know, the doors are open and people can–
PIPPIN: I just popped into the Mergebot.com website, and it looks really, really good.
BRAD: You just discovered the secret. It’s actually been soft launched for, I think, a week and a half. You’ve been able to — anyone that stumbles upon Mergebot.com right now can actually sign up, buy a subscription, and use Mergebot. Yeah, we kind of pushed it out.
Gilbert was going on vacation for two weeks, so we didn’t want to promote it and then have a bunch of people come to the site and not have Gilbert around, right? That would have been kind of a silly thing to do, right?
PIPPIN: You had a pretty long beta period. I know you’ve talked about it numerous times over the last 10 to 15 episodes or so. Was that beta period helpful in taking care of a lot of those complex problems?
BRAD: It helped a little bit. Not as much as I would have liked. I’m not sure what the problem was there. We had over 100. I can’t remember the exact figure. But, yeah, we had a pretty good number of beta users, but didn’t get a lot of feedback and didn’t get a ton of use. I feel like if we had had a higher price tag on Mergebot — the price was only $9 a month, right? A lot of people, I think, were just not using it and didn’t use it, so they couldn’t really give us feedback, right?
My hope is that now that we’re launching and the price tag is higher, if you sign up and use it, maybe then you’ll be able to tell us what’s not working for you and what needs to be changed. Maybe nothing needs to be changed. Maybe it’s good the way it is. But I don’t feel like we’ve gotten enough feedback so far to make a determination on that yet. Part of the motivation to launching is to get more people using it, get more feedback, and hopefully shape, you know, push Mergebot in whatever direction it needs to go in.
PIPPIN: Tell me about the system, app.mergebot.com, and the registration system you’re using. Is this completely custom? Is this a platform? What is this? It’s obviously very different from your Delicious Brains setup that a lot of people are used to. What is this?
BRAD: Yeah, it’s Laravel Spark. Laravel is a PHP framework, but then there’s this kind of package that you can install with it. The guy that’s behind Laravel itself, he also wrote this, and he sells it. I remember I paid $99 for it. It was ridiculously cheap. Yeah, so you buy a license for it, plunk it in, and it gives you user management. It integrates with Stripe for your billing. It’s basically software as a service in a box kind of thing.
We have a few complaints about it here and there. But, overall, it’s done a great job for us because it saved us loads of time not having to build all that stuff, as you know. It’s a huge time suck to have to build your own user management and e-commerce stuff. Oh, my God. Yeah, it’s been great, overall.
PIPPIN: Super cool. I’m really excited to hear in our upcoming episodes and get updates on how the launch goes.
BRAD: Yeah. Yeah. Something that we spun out, so one of the things that we had to build for Mergebot was an editor for serialized data. Let’s say you edit a widget, just a WordPress widget on your dev environment and you’re using Mergebot, and someone edits that same widget on the production site, well, you’ll end up with a conflict because you both edited the same widget, and it’s in the database. It’s just stored as a serialized string, and so it shows up in Mergebot as a conflict.
You have to be able to visualize what has changed in that string and then edit it to determine what you want it to deploy as. Maybe the person changed the title on the production site, but you actually, when you deploy, you want to change it to something different. In Mergebot you can actually edit that serialized data, and it’ll handle re-serializing it and everything.
What we did is we spun that out into its own little app. If you check out SerializedEditor.com, you can actually paste in some serialized data there and edit it, and then get the serialized output of the edited version. Yeah, I’ve already used that. I used it the other day, and it’s so handy. It’s incredible.
PIPPIN: I don’t know how many people have tried to manually edit serialized data. It’s painful, ridiculously painful, and most of the time you just totally fail and it doesn’t work. I saw this when you guys announced it, I think on Twitter, a few weeks ago and just immediately made my heart sing. This makes me so happy. This is just one of those little tools that you don’t need it that often, most likely. But when you need it, it’s just perfect.
BRAD: Yeah. Yeah, exactly. I got way more excited after I used it. I had to go to Twitter and say that I just used it, it’s so awesome, because, yeah, otherwise if you had to do it manually, oh, man. What a pain in the butt.
What else have you been up to, Pippin?
PIPPIN: I mentioned that we had several big updates in the last few days, and so the second one was the 2.8 release of Easy Digital Downloads of the core plugin. We made kind of a commitment earlier this year of trying not to add new features. It’s so easy in product development to just add new features, add new features, add new features. We’ve always tried to be a little bit more reserved on what kind of features we had. But look; we’re human, and it’s very easy when you start to see request after request after request for certain features to just screw it and say, build it.
Well, we kind of recommitted ourselves to being a lot more cautious with what we add and to put a couple of releases together that focused purely on refinement. Our previous Easy Digital Downloads 2.7 release was the first of these. That one focused entirely on just trying to polish things. Make the experience better. So it made a lot of improvements to various screens. It improved some inconsistencies, et cetera.
Yes, there were still new features added, but the goal here was still to make improvements to the existing system and solve pain points as opposed to building new features. Well, EDD 2.8, which was released yesterday, was in this same vein. Sometimes this means that it looks like a lackluster release, but this is actually — these kind of releases are my absolute favorite, I think, because it gives us an opportunity to kind of go back, find those nooks and crannies that aren’t very good or find those eyesores that you’re really tired of looking at and fixing them.
We had one particular eyesore that has bothered us for years. We had this feature in EDD called Variable Prices. What it really is is that when you create a product, you can have a single price on a product, or you can have multiple prices. For example, let’s say that you want to sell a product that has three price options: a basic license, an intermediate license, and an advanced license, or maybe bronze, silver, and gold. Then you want to have a different price point for each one of those. Then if you also happen to be selling subscriptions or license keys, you might also then need to say, okay, this one recurs monthly. This one recurs yearly. This one recurs yearly, but has an additional sign up fee. This one has a license activation limit of one, and then this one is up to 5 sites, and this one is up to 30 sites, et cetera.
Those kind of options for price options for the variable prices feature really had a terrible UI, because when variable prices were first introduced, it was this repeatable table row where you have two input fields. You have the name of your price option, say bronze, and then the price of your price option. That’s it. There are two fields.
Well, as we add more features to EDD core and to various add-ons, you suddenly get a third input field. Maybe an option to set what is the default price option that is selected when the page loads. Then you get an option — you introduce a feature into your licensing plugin that allows you to sell a lifetime license.
Then you introduce a recurring payments plugin that allows you to say, is this price option a recurring option: yes or not? Okay. How often does this recur? How many times should this be billed? What is the sign up fee?
All of these options, because the original design used a table, all have to fit within one table row just due to the constraints of the original design. There’s a blog post in the show notes that you can see a screen shot of, but basically what ended up happening is all of these options just got crushed together on smaller screens. If you had anything smaller than, say, a 22 or 24-inch monitor, this interface was just janky and jagged, and not good at all. It also meant that it was super inflexible for creating a better experience, like separating out your license key settings from your recurring payments settings from the default standard settings inside of EDD core.
Well, we decided to fix all of this. We went about it, and we’ve tried to fix this for two years now. There’s a reason that it was super tricky, and I’ll get to that in a moment. We first tackled it and said, look, let’s just take all of these options that we have on this screen and just design them into a layout that we like. Let’s just do that, and we’ll figure out how to manage backwards compatibility from there.
Here’s the kicker. There was no way to take that data. All it was was a table row and then a whole bunch of table cells. Every input option was basically a cell within the table.
Well, tables are super inflexible when it comes to your CSS layouts, so you can’t stack cells on top of each other. You can’t float one to the left, float one to the right, position one over here. No. They’re going to be in a line. That’s just the way that tables work because, well, it’s a table. That’s what it’s supposed to do.
We made this mistake of putting all this data in tables, originally. Why was this a problem? Why can’t we just get rid of the table? Well, the problem is that we had a lot of extensions, a minimum of five, that were very heavily used that all registered their own options inside of this interface.
Well, we didn’t have a formal API for how to add those options to the interface. Literally, adding an option to the interface was a do action call, and you just echoed your own table cell and your own input fields into the table. Then it just shows up. There’s no formal API. All it is is a do action just echoing out HTML.
We had two options. We either had to completely break sites for people that, if you update EDD core, but then you don’t update an extension, or maybe that extension hasn’t even been updated for compatibility yet. We didn’t want to have this broken HTML where you’ve got new HTML for core and you’ve got old, broken table cells being injected into a non-table now because an extension has been updated. We wanted it to be completely seamless and just work, even if they’ve never updated an extension.
We made it magically work. We built a backwards compatibility layer that detects. It looks at all of our actions inside of this area, and it rewrites all of the HTML on the fly.
BRAD: It strips out TD tags for, what, divs or something?
The fun thing now is that you can update to EDD 2.8, which does not use any tables for this interface, and you can be using an older version of software licensing that injecting tables into this interface, and there’s no tables on the page. They’re all gone.
It took a while to make it work, but it works seamlessly.
We looked at this with, okay, we have our extensions that we know about, and we have the ones that we control. We could just go ahead and update them all at the same time and push out updates simultaneously. If anybody experiences a breakage, we just say, “Hey, you’ve got to make sure that you update both of them.”
But what about all of the extensions we don’t know about? What about the developers that have done this for custom client sites? What if they’ve added their own options, which we know people have? We’re going to ignore those? No. That’s not an acceptable answer, in my opinion.
We said no, we have to fix it. We have to make every extension automatically work, even if they’re not updated. And so that’s what we ended up with.
That’s why this version of 2.8 is really exciting to me because it took that, took a challenge like that. It made the UI a whole lot better, and it made it a lot more flexible. Even if your extension is using the old HTML structure, it still gets rewritten into the new structure, and there is now a more formal method for how you’re supposed to register your settings and how you can for anybody that wants to use it going forward. That was a pretty fun little challenge.
It was one of those ones where you have — you get working on it, and you’re trying to figure out how to solve it. Then you get completely stuck, and you just kind of throw your hands up like, well, we’re screwed. We’re doomed. Then you feel like you have a little epiphany of, wait, I can do this. And it worked, so it was pretty cool.
BRAD: Very cool. Yeah, it looks great, by the way. It looks much, much nicer now.
PIPPIN: It’s pretty fun to play with it knowing behind the scenes that that plugin totally tried to output a table and it’s not there. It is not on the page.
BRAD: Yeah. Yeah, that’s really cool. Well done.
PIPPIN: You guys have been working on a couple of updates, I believe, for a long time now with Migrate DB Pro. Is that right?
BRAD: Yeah. Yeah. We’ve kind of been struggling a bit. We started about nine months ago on this release. It definitely goes against our target this year. It was to release at most every four months, and so we’re way over that now. It’s totally my fault.
This release of Migrate DB, the focus of it is the import feature. Migrate DB Pro is going to have an import feature. You’re going to be able to import an SQL file. That’s your migration. It’ll replace–
PIPPIN: You know that would have been great if you had that last week when I had to import my export file from WP Migrate DB Pro. No, that’s awesome. That is a great, great feature.
BRAD: Yeah. Matt, who has built that feature before for the plugin that he built and we acquired, Better Search Replace, he was working on that. But basically I was thinking, oh, Matt has built this before. I’m just going to give it — just say, “Matt, go build this,” and that’ll be enough. Right?
That’s what I did. I didn’t ask him to scope it out and work with us together to figure out what’s the kind of MVP, what’s the minimum viable product here, what’s kind of the smallest scope we can do for the first release? I think that’s why we’re nine months in now because we didn’t scope it out in the beginning, so the scope kind of just ballooned gradually. It was not managed right, so that’s why. It’s actually one massive pull request, so instead of scoping it and doing a very little bit and then doing a pull request, merging it in to the master, like, we tried to do it all in one big PR.
The good news is that we’ve learned from this and this shouldn’t happen again. I’m certainly not going to make assumptions and just say, “Here. Just build whatever.” That’s silly. In retrospect it seems really dumb that I did that. But hindsight is 20/20, so what are you going to do? As long as you learn from it.
PIPPIN: Yeah, I think that we’ve made that same mistake a few times, both with just a, “Hey, just go build this,” without much scope or anything laid out, and then we’ve also made that same mistake where you get everything in one big PR, and then you realize we literally cannot. You can’t take this out without removing everything, and you cannot finish a release until this is done because it’s all together. Piecemealing things, we’ve learned that the hard way a couple of times that, oh, my gosh, if you can do things in smaller chunks, it saves so much headache in the future.
BRAD: The other mistake we made was assigning the whole thing to one developer. We’re going to try more of a collaborative approach. Even though something looks like a unit and is not really fit for more than one person, we’re going to really try and have two people work on it anyway. That’s something we’re going to experiment with in our next release.
The fear is that they’re going to step on each other’s toes and there’s going to be a lot of merging that’s going to be needed. But the alternative is that one person toils away for a really long time before they actually have something to show, right? Then there’s the whole bus factor, right? If that developer gets hit by a bus, and then no one knows what the heck they were working on and that kind of thing.
Is that something you guys do? Do you try to — I mean obviously you’re going to try to break things up as much as possible. But when you have a chunk that’s pretty large, does it just go to one developer still?
PIPPIN: It depends on the project. I think we’re probably better about not putting it on one developer in EDD than we are the other two projects. I think some of that is just because EDD being — EDD is where we started having a collaborative environment first. We’ve had it as a collaborative environment longer than either of the other two, and so naturally we’ve adapted to that better.
We have two developers on Affiliate WP. We have two developers on Restrict Content Pro. Then we really have two full developers on EDD. We really have two on each.
I would say most of the time it’s one project per developer, but I try to encourage that there is continual review and assessment from each other. Okay, so here is one of my number one rules for our team is that nobody should ever be working in a vacuum. I don’t care if it is a tiny thing or a giant thing. If you are working in a vacuum, it is probably one of the easiest ways for us to get into trouble.
Okay, so we’ve had an example where we had a feature being worked on for months. We pretty much had to throw it all away because it was being done in a vacuum and it came out, and it wasn’t like it was supposed to be. It didn’t work how we needed it to work. There was total lack of communication, and it was really my fault for not managing it properly. But it did happen.
If you have that lack of communication, it’s very easy for that to happen. No vacuums, that’s my number one rule now.
BRAD: What about Restrict Content Pro 2.9?
BRAD: Let’s hear about that.
PIPPIN: That’s our other major update. This one, it’s actually a really big update. It has a ton of stuff. But it did have a couple of — it had one very significant challenge. Mostly this is another polish and new feature release. Really, we’re getting in and trying to add in some of those features that we’ve needed to have in for a long time.
For example, we now support notifying subscribers before their subscription renews. Give a subscription courtesy reminder. That’s something we had never had. We now have that.
We added in an option to restrict complete post types. We would have a lot of people that would come to us. They would have a site that they want to set up a membership on, or they already had a membership, but they had a big catalog of data, and they want to say, “Well, every tutorial in my tutorials post type needs to be restricted and the bulk options that you guys offer don’t work for me.” And so they wanted the ability to just say, “This post type is restricted.” We’ve now added that.
Then we’ve also had this big limitation for a long time with user roles. We’ve always supported restricting content to user roles, but only the default user roles within WordPress. If you had a custom user role, it didn’t work. We’ve now fixed that, and you can now use any registered user role, and you can restrict content to any of them.
There are a couple of others, little improvements, but there was one major one. It’s kind of funny. This release is actually about two to three months behind when it was supposed to go out. We had started this as, hey, this is going to be our one month turnaround release. We’re going to fix a bunch of little things, just do some quick polish, and ship it. Three and a half months later, we’re not shipped.
What happened is we have a payments database. Thankfully, this one already uses custom tables, so there’s no migration there. But it had some limitations. It had — it never — it didn’t have any way of handling, say, a pending payment record. It only kept track of completed payments and refunded payments. But the idea of having a payment that has been started, but is not completed, didn’t exist in RCP.
It turns out that adding support for that was really, really tricky when you’re dealing with a large, historical data set because then basically we discovered that we had to add in a huge backwards compatibility layer that says, okay, what if you access an old payment record that doesn’t have a status set on it? What if you tried to insert a payment record from a custom gateway that somebody has developed that doesn’t know that you’re supposed to be storing a payment status now? What is the default status? Is there a way that we can detect it? How can we set it? What should it be set to?
We ended up discovering a lot of those kind of edge cases where we had to support people that have used custom gateways because there are a lot of custom gateways available for RCP, and a lot of people have built their own for client sites that we don’t know about. RCP has been around for almost seven years, and so there’s a lot of sites that have been running for a long time, and we had to make sure that all of those continued to work as we were making these changes to the database tables. And not only that their data was still good, but that the old way that they were creating the data, a.k.a. the payment records, was still valid.
We actually did a couple things. We added the support for pending statuses. We added, like we now — this seems really simple that we should have done this, but we never used to keep a record of when you used a discount code on a subscription. It seems kind of silly, but we didn’t. That’s now used.
Our payment records used to be stored with a subscription name. Let’s say that you have bronze, gold, and silver as your subscription levels. We would store the name of a subscription level as opposed to the ID of the subscription level, which meant if you ever went and changed the name, it broke the relationship. That was annoying. That’s now been fixed.
There were a few other things. Really, the RCP 2.9 release, there are three really nice new features, but then they did a ton of polish. I think we had a total of like 24 bugs and 19 distinct improvements that we made.
Now that it’s out, we’re really, really confident that it’s stable. Actually, we haven’t released it yet. We released a beta. Here in a week or so, we will release the final version of it.
We’ve been busy. I know you guys have been busy. Anyway, we’ll have probably quite a bit more to share next time here. We’ll probably have another — I’ll probably talk about database migrations for the next four months, just to forewarn you.
BRAD: Yeah. Nice.
PIPPIN: All right.
BRAD: Yeah. If anybody is interested in sponsoring the podcast, you can check out ApplyFilters.fm and click on the sponsorship link. We’ve got all the info in there for you. Yeah, we’ve got sponsorship spots open.
PIPPIN: We’d be more than happy to work with you. If you have a WordPress product, a WordPress service, or anything like it, let us know.
BRAD: For sure. Thanks, everybody.
PIPPIN: All right. Thanks for listening, everybody.