INTRO: Welcome to Apply Filters, the podcast all about WordPress development. Now here’s your hosts, Pippin Williamson and Brad Touesnard.

PIPPIN: Welcome back to Apply Filters, Episode 60. Today, Brad and I are going to go over some of our recent updates. But then, more importantly, we want to talk about the problem of database merging. We’re going to cover that in depth. First–

BRAD: This episode is sponsored by Gravity Flow, an advanced add-on for Gravity Forms that allows you to automate your business processes, whether you need to set up workflows for purchase orders, job applications, admission forms, project initiations, vacation requests, or any other kind of workflow that involves advanced feedback loops, approvals, or anything like that.

Gravity Flow allows you to do this easily by leveraging the power of Gravity Forms. You can find out more at GravityFlow.io. Once again, thanks very much for sponsoring this episode.

PIPPIN: Thank you, for sure. All right, Brad. Do you want to start us off by telling us a little bit about what you guys have been up to in the last couple of weeks?

BRAD: Yes. Well, in the last episode I mentioned that Migrate DB Pro 1.6 would be released when that episode was published, but it actually wasn’t because we ran into a kind of last minute bug. It was a scaling bug, so when you would try to migrate with, let’s say, 10,000+ media files, so if you had 10,000-attachments-plus in your media library. If you tried to migrate a site with that many attachments, it would add them all to the DOM and your browser would start to get either really slow or crash depending on how powerful your laptop is. That was a problem, so we had to–

PIPPIN: Just a minor problem.

BRAD: Yeah. Yeah, so we had to not add all of the attachments. We’re talking just about adding list items, just the file name to the UI, but 10,000 file names. Yeah, we stopped doing that. Now I think we add, like, 100 and then we just have a note at the bottom that the rest are hidden. That’s done, and we ended up releasing that version on April 25th instead.

PIPPIN: I’ve got to say the new UI is pretty.

BRAD: Awesome, yeah.

PIPPIN: Yeah, the progress bars, the way that it shows the progress for each individual table, I’ve got to say I wish that I had a reason to use it right now because I want to play with it.

BRAD: Yeah. We’ve gotten lots of great feedback on it. Interestingly enough, our renewals have spiked because of it. I looked at last year’s major release, version 1.5, and there was no spike in renewals.

PIPPIN: Interesting.

BRAD: I think people like the shiny new.

PIPPIN: People like pretty things.

BRAD: Yeah, exactly. We’re going to try to do that in future versions more often. This is the first kind of UI update since we launched three years ago, so we’re going to try to do some UI refreshes.

PIPPIN: A bit crazy that it’s been three years.

BRAD: Yeah, I know. Right? It’s crazy. What else? We’ve been working on the next version of WP Offload S3. That version is going to be a big change to the way we store URLs in the content. Right now, when you insert an image, for example, into content, it adds the S3 or CloudFront or CDN URL in there. Then say you uninstalled our plugin. Well, those URLs would still be in there, so you’d have URLs to S3, CloudFront, or wherever.

What we’ve decided, and we were kind of inspired by the responsive images addition to WordPress and the way they filter the content to add the SRC set stuff. We were inspired by that and just decided: Why don’t we do that; why don’t we just filter the content and replace the URLs when the page generated rather than insert them directly into the content? That’s going to be a big change.

PIPPIN: That means you were doing it all on the fly, as opposed to you weren’t previously?

BRAD: Exactly. Exactly, so we’re going to be replacing the URLs on the fly rather than inserting–

PIPPIN: Now, does that mean that you’re only replacing the URLs, or you’re also handling the transmission of the file to S3?

BRAD: Oh, yeah. We’re doing both, for sure.

PIPPIN: Well, but the transmission to S3, is that when the image is first uploaded, or when the image is rendered on the page?

BRAD: It’s when it’s first uploaded, so when you drag and drop a file into WordPress, when it uploads to your server and then uploads it to S3, like right after. That’s how that works. It’s kind of like a little relay.

I guess the reason we’re doing this is because, when you change a setting, like one of the settings, maybe you changed the URL structure or something, it has to do a find and replace in the background. It just takes forever to do that. But, with doing this on the fly, you change that setting; it’s instant. Any else that’s generated on the fly, it’ll be the new setting.

The thing we ran into just recently is caching. There’s actually a really expensive query that we have to run, so you can imagine when you’re doing this on the fly, so you’re going through the content. You’re identifying a local URL that is referencing an image. Then you have to search for that image path in the database. We’re hitting the post meta table searching for that string, that kind of the file path string. Well, you know, Pippin, that querying full text on the post meta table is bad news, right?

PIPPIN: What do you mean? It’s like the fastest query there is.

BRAD: Right.

PIPPIN: Yeah, no. No, bad news for sure.

BRAD: Yeah, so what we decided is to cache that. This week, we kind of reassessed it. We were going to cache all the URLs and, like, what S3 URL that it maps to. But then we decide, oh, but then we have to purge that cache whenever the settings are updated and it gets real messy. So we decided just to cache it kind of at the lower level where the select is happening to define the ID for that because the chances that the ID is going to change is almost nil, right?

PIPPIN: Right.

BRAD: And we’re not caching it in a typical sense. We’re not using the WordPress object cache. We’re just going to cache it in, I believe, a transient. No, that’s not true. That was one of the ideas, but what we decided is to attach a cache meta, just a piece of meta data, to the post itself, so we can just look that up and we can find the URL in that cache array. It should be pretty performant and just avoid having to run those queries more than once – ever. That’s kind of one of the things we’ve been working on.

PIPPIN: Great.

BRAD: What else? We’ve been reorganizing our site’s theme. The theme has kind of just been thrown together. When we first launched, we were in startup mode, so we were just like “blah,” you know. It wasn’t pretty. We’ve refined it over time, but there was just a ton of logic code in the theme itself.

PIPPIN: This sounds very familiar.

BRAD: Yeah. Yeah, I mean you always neglect–

PIPPIN: I’m sure a lot of other people can relate to this as well.

BRAD: Yeah, well, I mean it’s not — your site is not something you release, so you can neglect it. You can do as many hacks as you want. No one will ever see them besides you and your team, so it tends to be the biggest patchwork of things.

PIPPIN: Absolutely.

BRAD: Yeah.

PIPPIN: We just recently did the exact same thing. We launched a new version of the EDD site about a month ago now. Part of that project was scrapping everything and starting from scratch to get everything organized, get our CSS in order, et cetera. That was a big project.

BRAD: Yeah. Yeah, it’s not too bad. It hasn’t been too bad so far. What we decided to do — well, first of all, one of the goals with the redesign is to be able to roll back to the old design instantly if we are having trouble with the new theme for some reason. The idea is to create two themes. But, to do that, we don’t want to duplicate all the code that’s running the theme.

We thought about moving it to a plugin, but that’s a little dangerous because plugins can be deactivated, intentionally or unintentionally. What we decided is move it to an MU plugin, which they can’t be deactivated. That’s what we’re going to do. We’re going to have most of our code in an MU plugin, and then we’ll have two themes that use that code. It seems to be the right way forward, but time will tell.

We’re hiring another developer. That’s the other thing we’re doing. We actually have someone on trial right now.

PIPPIN: Very cool.

BRAD: Yeah.

PIPPIN: You’re hiring almost constantly, it seems.

BRAD: Yeah. It takes us a long time to hire someone, though, so it’s kind of an illusion.

PIPPIN: True. Yeah, so even if you’re hiring this week, and you’re hiring in three weeks, it doesn’t mean that you added a new person between those times.

BRAD: Exactly. It means that we’re still looking.

PIPPIN: Yep.

BRAD: Yeah. What about you? What have you been up to?

PIPPIN: Last week, I was in San Diego for WordCamp San Diego. That was this last weekend, and that was a great time. The organizers there did a tremendous job, the same as last year. It’s a great location, awesome venue, really good selection of speakers and really good attendee list, so that was a good time. I was able to spend a few extra days out there. One of my team is out there, as well as several teams that we’ve done some work with–third party developers–and so it was nice to go out and get some face-to-face time with them.

BRAD: Right. Were you one of the speakers?

PIPPIN: Yes, I did a session on “Ask me anything about plugins,” so it was kind of an open QA session where I gave a quick background and told a little bit about what I do, what I’ve worked on, and then opened it up for any questions about anything related to plugins from the audience.

BRAD: That’s cool.

PIPPIN: It was a lot of fun.

BRAD: Did you have some questions primed to get things rolling? I found that, in the past, QA sessions, sometimes you open up the floor and it’s just like crickets, you know?

PIPPIN: Absolutely. I talked to the organizers and said, “Hey, I want to prepare for this and make sure that if we get crickets, let’s make sure that there’s somebody there to ask some questions,” and so I had a few people that I knew were in the audience prepared to ask something if no one piped up. But, we actually didn’t have any trouble at all getting questions. We had questions immediately, and we probably could have gone significantly over our time slot.

BRAD: Were they mostly about development or business? What were they?

PIPPIN: There were development questions, there were business questions, and there were user questions.

BRAD: Right.

PIPPIN: So, we had a nice range of everything.

BRAD: Right. Did you feel you got more of one than the others?

PIPPIN: Maybe more business questions related to my own experience, my personal history with plugin developments and people asking about some of that history or how we got from A to B. Business development questions, really, those were probably, I think, the most common.

Beyond that, we are just about done with Affiliate WP 1.8, which is an update we’ve been working on for quite some time now. It’s getting close to being done, and we’re hoping for a release next week.

Restrict Content Pro got version 2.5.3 shipped this morning. That fixed a slew of both minor and important bugs. There were a couple important bugs related to proration and multiple subscription levels, erroneous multiple subscription levels. For example, somebody could subscribe, and their previous subscription was supposed to be cancelled in the merchant processor, but it was left active, so then they have one subscription on the site, but they have two in the merchant account – a couple of bugs like that.

Then we also just pushed out an update to our Zapier integration for Easy Digital Downloads that adds a couple of new triggers for things like when a customer account is updated or when a file was downloaded, and also added a slew of new triggers for our recurring payments plugin. Now, for example, there are triggers for Zaps such as when a subscription is created, a subscription expires, it’s cancelled, it’s marked as completed, it is marked as failing, et cetera. There are a lot of automation tasks people can now do that are running in a subscription site through EDD Recurring Payments. For example, you could sync all of your subscriptions to a Google doc. Or, every time a subscription is renewed, do some kind of action. Or, if maybe a subscription is marked as failing, trigger a custom workflow to set up a follow-up to that customer. Lots of different things that you can do, so that was a fun little update that we got out, pushed out today.

BRAD: Zapier is just so powerful. It’s amazing.

PIPPIN: Oh, man, it’s so good. This morning, we were actually using it, setting up a new, internal workflow for us. We’re trying to get a little bit more active with sending out swag to high value customers, and so we’ve set up an automated workflow now that any time a customer passes a certain threshold that we add them into an internal Trello board for sending them a thank you, basically. We’re using Zapier for that, as well as the new update that we pushed out for EDD, the Zapier integration. Amazingly powerful what you can do with Zapier. It’s so cool.

All right, let’s jump into the meat of this.

BRAD: Sure.

PIPPIN: We’re going to talk about database merging.

BRAD: Before we do that, actually, there’s a little follow-up on last week’s episode about the full screen UIs thing. I mentioned that I’d love to see a plugin be more like the customizer and just take over the entire dashboard of WordPress and just have kind of a back button to get out of it. Apparently there are quite a few plugins doing that now. People commented on our podcast episode, and I think I got a couple tweets as well.

But, anyway, there are a couple, so Coming Soon Pro does this and the new WP Forms, a form builder, does this, and–I don’t know how to say this–Kanban.

PIPPIN: I think it’s Kanban.

BRAD: Okay. Kanban for WordPress, that does it as well, so there are definitely lots of plugins out there doing this now, which is cool to see.

PIPPIN: We want to talk about the database merging problem. This issue is basically when you have two databases and you need to merge them together in some way or other. Mostly, this is going to be me asking Brad some questions and Brad telling us about something that he’s been working on. Do you want to start by giving a little bit more of an overview about the problem of database merging?

BRAD: Yeah, so I’ll just take you through an example. Let’s say you’re working on a new client site and the client has an existing WordPress site that’s live. And so what do you do? You make a copy of it. Copy the files. Copy the database. Put them on your local machine. Set up a dev environment and start coding away and updating the data in that database, in that local database.

Well, while you’re updating the data in that local database, the data in the live database is also changing. You’re changing data in local, the data is changing on live, and then, at some point, you’re like, “Okay. Now I’ve got to get this local database up to the live site, but I can’t overwrite the live site because then I’ll blow away all the changes that have been happening up there.”

Therein lies the problem. You’ve got to somehow get the changes from your local database up to the live database without overwriting it.

PIPPIN: I’ll tell you this is a problem that is so much more problematic when you step into the world of e-commerce. Just to give a quick example: Imagine you have a store and then you have a development version of that store. And so, when you start your development, you sync everything over. Let’s just say you have an hour period where you’re going to make some changes on your development server. Then you’re going to deploy them to live, and you need to deploy those database changes as well.

Okay, so what you need to do is you need to be able to make all those changes on the staging server and then deploy them to live. However, in that time, let’s say it’s an hour or 2 hours or 24 hours. It doesn’t matter. You still have orders processing. It’s especially a problem for very, very active stores.

And so, you now have orders that are processed on the live server that are not on the staging server, and so it is impossible then to merge that database over. Not necessarily impossible, but very, very difficult. As a store, you don’t want to take your live site down for 2 to 24 hours in order to make those changes.

BRAD: Yeah, exactly. Well, first of all, when everyone gets to this point, they realize, “Crap!” Then they start searching Google for some kind of magic database merging tool that you can just give it–

PIPPIN: Please let me click a button and be done.

BRAD: Yeah, exactly. But, there is no solution. Well, first of all, if you get into that situation and didn’t see it coming, very likely you’re just in big trouble because even if a tool like that did exist, it would still need the snapshot of your local or the snapshot of the database when you moved it to local so that it could compare that with your current local database and determine the changes, right? Because you have two databases that have completely diverged, it doesn’t have the snapshot of when you did that initial migration, so it has no reference point. That’s a problem. When people get into that situation, they usually just start over and just start making the changes again, like, over, you know, on the live site.

PIPPIN: Guilty.

BRAD: Yeah, so it’s bad, bad news. But, generally, people don’t get into that situation again, or they might do it a few times and then they figure a way around it.

The way around it that most people use is they just keep a notepad, either a digital one or one beside their laptop, and they just write down what they do. For example, if I go in and change or maybe I add a bunch of taxes to the e-commerce system, tax rates so that I can test them locally and do some development around them, well, that has to be done again once the site is deployed. So, I’ll just write down a note that says, “Add taxes.”

PIPPIN: Here. I’ve got another example for you. This is one that we had recently. We launched a new version of the Easy Digital Downloads website about a month ago. With that change, we rebuilt a whole bunch of different pages. We used different short codes. We used different template files, et cetera. We had them all on staging, but we had to make notes of every single page that needed a change because we weren’t going to be deploying the database from staging to live. We were going to deploy the code, but not the database.

BRAD: Exactly. That’s very interesting how you guys are currently doing it right now. That’s really interesting.

PIPPIN: We’re doing it the easy, lazy way.

BRAD: The way that makes — I mean the way that works, right?

PIPPIN: It’s the way that works.

BRAD: The way that works, yeah. So, the other way of doing it would be to script everything. Like in your example, you could do a wpinsertpost or wpupdatepost, or whatever, and actually have a script that adds the content to that page so that you could just run that script at deployment time and it would just kind of add all that content or update all that content on your live site, so in kind of one shot. The big advantage to doing that, I mean that takes a lot of effort to go through and script everything. But, the big advantage is that you can test your deployment.

For example, what you could do is you could replace your staging site with the live site, with the current live site, and then apply that script to the staging site, and then test it. Then test that staging site to make sure all of the script changes have been applied properly and that the site is working and all that stuff. Of course, you could continue. You could do that multiple times before you actually deploy to live so that you’ve tested your deployment a bunch of times before you actually do it to the live site.

That’s what a lot of people do, but, like I said, that’s a lot of work to script things, especially if it’s something like a plugin setting, like if you’re updating a plugin setting. Plugins store data. Everyone does it differently, pretty much. Some might store an option for each setting, or some just stick all the settings into one option as a serialized array. So, if you’re going to script that, you’re going to have to figure out how the plugin is storing its data and then script, emulate that, essentially. That’s why people don’t generally do it because it’s a lot of work.

Have you ever scripted, done any scripting for deployments before?

PIPPIN: Not related to the database.

BRAD: Right.

PIPPIN: No. I’ve always kind of just avoided those. One of the other problems that comes up really frequently is let’s say that your staging site, it’s not just out of sync with live in terms of maybe new orders or new blog posts that have been written. It’s not something I’ve ever really tried. It’s a problem that I’ve run into numerous times and wished I had a better way to do it, but not a challenge I’ve ever decided to undertake.

BRAD: Well, speaking of better ways to do it–

PIPPIN: All right, let’s hear about this.

BRAD: We’ve been working–

PIPPIN: I’ve heard rumors that you’ve been working on something.

BRAD: Yeah.

PIPPIN: Can you share some previews with us?

BRAD: Yeah, so we’ve been working on this solution, and we haven’t really been keeping it secret. We tweeted a little bit about it. I think we mentioned it in a few blog posts. We haven’t been super upfront about it, but we haven’t been keeping it a secret either.

It’s called Data Hawk, DataHawk.ioMergebot.com. You can go there now, and there’s a landing page where you can sign up. Just add your email address to our list and you’ll get news about it.

Here’s the thing. It’s the solution to this problem we were just discussing. What you do is you install our WordPress plugin on your local site and on your live site, both. Then, as you’re making changes locally, it’s going to save those changes, basically record those changes to our cloud app. Then, the cloud app basically allows you to pull those changes to the live site and apply them to the live site. That’s pretty much it. That’s it from a high level, very simplistic view.

PIPPIN: Okay. I have a few questions for you about it.

BRAD: Sure.

PIPPIN: First, that sounds awesome.

BRAD: Maybe I should go through the flow, the development flow that we’re trying to get people to do.

PIPPIN: Let’s do it.

BRAD: We’re trying to emulate what I just told you about the scripting. The advantage with scripting is that you can test your deployments over and over again. What we’re thinking the ideal flow for developers would be, so you copy down the live site to your local, replacing local. Do some development. The changes are recorded up to our app.

Then, at a certain point, you’re like, “Oh, tons of changes have happened to the live site in the meantime, but I’d really like those in my development environment.” What you do is you just replace your local with live again, but then you can apply the changes from our app, from our cloud app. You can pull those changes down and apply them to your local dev environment and basically keep going where you were, but with the live changes in place as well.

Another thing that happens there also is it detects any conflicts and allows you to resolve those. What we want is we want you to keep doing this. We want you to keep refreshing your local database and applying the changes to it and basically testing the deployment over and over again so then when it comes time to pull those changes into the live site, you’ve done it a bunch of times already. So, you shouldn’t feel squeamish about it at all. It should be just another day at the office instead of another 3:00 a.m. at the office. That’s kind of what I’m hoping developers will adopt, that kind of flow.

PIPPIN: Yeah, that’s stellar. If I can, I’ve got a couple technical questions for you on how it works.

BRAD: Yes.

PIPPIN: First of all, will you be able to pick and choose changes, or is it all or nothing?

BRAD: Right now, we don’t have that ability, but we’ve already mapped it all out. I think we decided that it will be in the first public release, but it may not be in the first public beta. It’s on the list. It’s tricky because of dependencies, so what we’re doing is recording every query and parsing each query, parsing the data that’s in those queries, and figuring out the relationship between each query.

Let’s say you de-select one query that you don’t want to deploy. You say, “Ah, I don’t want that one,” so you de-select it. Well, if there is any child data associated with that, for example inserts into post meta, so say you de-select. I don’t want this post inserted anymore, so you de-select it. Well, you have to actually de-select all the post meta as well. Otherwise you’ll just end up with a bunch of junk in your database.

PIPPIN: Right, a little orphaned rose.

BRAD: Yeah, so it’s actually a bit tricky, which is why we haven’t done it for the MVP.

PIPPIN: Makes sense.

BRAD: Or, I guess, private betas. I should be calling it private beta. Most people don’t know–

PIPPIN: Can you tell me a bit about how you’re actually tracking changes to the database? For example, tracking changes to any of the standard WordPress tables is relatively simple, whether it’s through the internal APIs or through the database class itself. What about plugins that use custom tables or things like that? Have you done the tracking in a way that you can identify changes at any part of the database?

BRAD: Yeah, so what we’re doing is we’re recording queries, so in the WP DB class, there’s a hook in there that basically you can capture any query that goes through WordPress. We use that to capture, make sure that we capture everything that goes into the database from WordPress. That being said, if some plugin is using PHP’s MySQL functions directly for some reason–

PIPPIN: Too bad.

BRAD: Well, yeah, too bad is right. We can’t cover that. Sorry.

PIPPIN: Right.

BRAD: That’s a very strange case; I will have to say. That’s what we’re doing. We’re recording everything. Right now, in the private beta, we’ve decided just to limit things to the core tables just because we’re trying to get this thing working. But, in the future, we’ve made it flexible enough so that we can feed the app data, so schema data.

One of the problems with the WordPress database is that there is no relationship information in the database itself. There is no way to tell that the user_id column in the post meta table, or the user meta table, actually relates to the ID table in the user’s table. We know that because, just looking at it, it makes sense. But, there is no data in the database that says that. There is no relational data.

Unfortunately, we have to map that in PHP. We have to say, you know, user meta table column user ID actually maps to the user’s table ID column. We have to put that in an associative array in PHP. The plan is to do that for every custom table, and the reason we can do that is because it’s an app and everyone is putting data into it. So, when enough people put the same data schema in, we can say, “That’s a common schema. Let’s approve that.” Then, in the future, it’s just approved for everyone.

We’re also going to add the ability to add your own schema. So, if you’ve added a custom table or a custom set of tables, you can define the relationships between those tables.

PIPPIN: Cool.

BRAD: Provide those to our app, and then our app knows what’s going on.

PIPPIN: I like that you’re crowd sourcing data schemas.

BRAD: Yeah. Yeah, when we first set out to tackle this problem, we really weren’t thinking of making a cloud app. The more we talked about it; I think Gilbert brought it up early on about making it a cloud app. I think I just dismissed it, of course. Probably silly idea, and then, of course, I just thought about it some more and started to see all of the opportunities where we could do really cool things if this thing was a cloud app.

For example, one of the things I want to do is allow teams of developers to work on the same deployment. So, like, if you and I were working on the same deployment, we could push our changes up to the cloud. And, like, say if you changed the title of a page to something, and then I also changed it, but I changed it to something different, the cloud app will know that and it will alert us, like, “Whoa. You guys have a conflict. You both changed this thing. Whose do you want when you deploy?”

PIPPIN: GitHub-esque merge conflicts.

BRAD: Yes, exactly.

PIPPIN: That’s cool.

BRAD: Then it will provide an interface for choosing which one you want to use on deployment. It actually has conflict detection for the live site as well. This actually works right now, so if I edit something on my local dev environment, so say I changed the title of the page, and someone changes the title of that same page on the live site, well, when I go to deploy to my local, it’ll say, “Oh, hang on. You’ve changed this title, but someone also changed it to something different. Which one do you want to use at deployment time?” This avoids the obvious problem with me changing things locally and then overwriting the client’s changes when they deploy or when I deploy the site and them getting very angry about that.

Obviously a title change of a page, that’s not a big deal. But, let’s say I changed the price of something when I was testing something and they changed it to something else or something like that. That would be probably much worse. That’s kind of the idea. It’s been very challenging.

Probably the biggest challenge was the IDs, the ID conflict thing. Do you know what I’m talking about when I say that?

PIPPIN: I think so.

BRAD: Yeah.

PIPPIN: Go ahead and run through it.

BRAD: Let’s say the live site has added a bunch of posts since I’ve created my local site. Then I’ve added maybe one post, let’s say. Well, the ID of that post locally, right, let’s say it’s 33. Well, there’s already an ID of 33 up on the live site, so I can’t use that 33 when I deploy.

The app actually manages all of that. It stores that change and then, when it goes to deploy, it’ll create a new ID, but it’ll also change the ID of any records that use that ID. For example, let’s say it’s an attachment. The attachment is attached to a post. That’s probably a bad example. What would be a good example?

PIPPIN: I could give you an e-commerce example because it’s one that we see in EDD a lot.

BRAD: Sure.

PIPPIN: This is assuming that the e-commerce data is stored in — well, in this case it is stored in post meta, in posts, but it could be custom tables and the same issue would still apply. Let’s say that you have a purchase record, and you have products. You need to make sure. If you sync the database, and let’s say that you have an order ID of 575, and you have a product ID of 600, so that 600 product ID is stored on that purchase record. Well, what if, when moving between this live and development site, the product ID changes, so if it was originally some number, it has to not only be updated in the product table, but also any order record that contains that ID, in order to maintain the relationship.

BRAD: Yeah. That problem was huge.

PIPPIN: That was something that we actually ran into recently because, actually, we see it a lot. We see people want to migrate order data or product data from one site to another site. And, in the extreme cases, which is actually not very extreme because it’s pretty common, people want to move purchase records, products, license keys, customers, subscriptions, et cetera, from one site to another. Most of the time it’s actually moving them from one site and putting them into another existing site. It’s not just cloning a site or anything. That is really, really tricky, maintaining all of those data relationships because of the ID problems.

We did it ourselves. We moved my Restrict Content Pro plugin. It used to be sold on PippinsPlugins.com. Now it’s sold on RestrictContentPro.com. Well, when we made that move, we wanted to extract all of the purchase data from the Pippins Plugins site and put it onto the new site, bringing over purchase records, license keys, customers, download logs, everything, and making sure that those relationships stay. It’s tricky.

We actually ended up writing a custom CLI command to do it so that it would pull a record out. It would map all the IDs, and then be able to put them back in. Each time it created one, it would go through and say, “Okay, now I need to update this ID to this because this is what the new one is.” It was pretty extensive. I mean it works, but it’s not a simple cut and paste.

BRAD: No, not at all. I think we’re about a few weeks away from a private beta. Yeah, if you’re interested in trying the beta, just go join our mailing list, DataHawk.ioMergebot.com, and I’ll send out an update soon enough with anyone that wants to try it out. Obviously we’re going to be sending out invites. We’re not just going to open the doors, open the floodgates, and let everyone in, because of scaling issues. It’s still early on. We’ve just got this on one server, so we can’t accommodate a ton of people at once.

PIPPIN: Right.

BRAD: Yeah.

PIPPIN: Do you have anything that you can share in terms of potential pricing for it, or is that still either private or undecided?

BRAD: Completely undecided. I’ve got some ideas, but it’s just really early days for that yet. One thing that I’m pretty sure about, though, is that people who sign up for Data Hawk will get a license for Migrate DB Pro as well. The reason that is, is because they both work really well together. It just makes sense for people to kind of get both when they sign up for Data Hawk.

PIPPIN: Yeah, that makes sense, or as a package.

BRAD: Yeah, kind of like a package deal. Yeah, I don’t think it’s going to cost any less or any more. We’ll just include it.

PIPPIN: Well, you know one of the cool ways that they work together is let’s say that I set up a new live site and then, okay, we launch. Now I want to convert. I want to make a new development server for all future development, so I go ahead and use WP Migrate DB Pro. I pull everything over, and then I continue to work from there and off we go.

BRAD: Also, Data Hawk has settings and stuff in the database, in the WordPress database. Migrate DB Pro is aware of that, so it ignores all that stuff, so it won’t overwrite your local database’s settings when you do a pull. That’s one of the big reasons we want people to use Migrate DB Pro because, if they just do an expert through PHP, MyAdmin, or the command line, they’re going to have all that data, all the settings from Data Hawk that’s sitting on the live site. It’ll basically blow all their local data away.

PIPPIN: Oops.

BRAD: Yeah, so it’s better. We’re still going to accommodate that situation, but it’s definitely better if they use Migrate DB Pro.

PIPPIN: Very cool.

BRAD: That’s the plan.

PIPPIN: For anybody who is listening that wants to follow up with any questions if they’re interested in not just maybe following up with the beta, but if they have questions they can ask you before further announcements, is there a best way for people to get in touch with you, or just be patient?

BRAD: That, for sure. I’m planning to put out a blog post and send out emails very soon. I’d say, just sign up to the mailing list. It won’t be long before you get an email in your inbox about it. Then you can just reply to that email and ask any questions you want.

I mean everyone is always welcome to hit me up on Twitter, @BradT. We’ve got an email address too, so [email protected][email protected], if you want to reach out to us if you have any questions about it. It’s whatever. We can comment in the show notes, too, on ApplyFilters.fm. That works too.

PIPPIN: Well, very cool. That’s a very exciting project. I know that, as a product and a service for everybody else, it’s something that–assuming it works well, which knowing your guys’ track record, probably will–solves a huge problem. I know that I can speak for myself, and probably many others, and say I’m pretty excited for you.

BRAD: Yeah, man. We’re very excited about it, and we just can’t wait to get people trying it out and kicking the tires.

PIPPIN: Awesome. Well, hopefully here in a few more weeks, a few episodes, we’ll be able to talk a little bit more about it, maybe after I’ve played with it a little bit after it’s up and available.

BRAD: Yeah.

PIPPIN: I’m looking forward to it.

BRAD: For sure, man. Sounds good.

PIPPIN: Well, should we wrap up here?

BRAD: Let’s wrap it up. Another shout out to our sponsors, Gravity Flow, GravityFlow.io. Check it out. Talk to you next time.

PIPPIN: Great. Thanks, everybody.