Issue with some previous youtube embeds

neptronix

Administrator
Staff member
Joined
Jun 15, 2010
Messages
17,528
Location
Utah, USA
Our forum is not showing youtube embeds from the past due to a database change from a plugin i attempted to install.
Current youtube embeds are safe to make and will work into the future.

I am working on a fix right now, which requires me to make large sweeping database changes.

Here is some corny hold music for you to listen to while i work on this.

[youtube]AC7gi1eVsco[/youtube]
 
Apparently some old youtube embeds work and others don't.

Clicking 'edit' and then 'save' on a post will fix the ones that are busted in the meantime.
 
Okay, it looks like changing the youtube embed settings ( unexpected side-effect of installing advanced editor ) jostled phpbb into embedding things in a new way which is incompatible with future exports of our forum. The phpbb software wanted to do things this way but did not make the change through successive upgrades we've made.

phpbb has also historically stored youtube embed data in various different ways over the years.
That will have to be reconciled with some database-fu in the future.

I would restore parts of our system from backups, but many changes were made last night which i'd prefer not to revert and lose. The process would likely result in an even bigger mess than current.

A fire is lit under my ass to work on the new forum format starting today.
 
FWIW, stuff like that is why I would never work on the "live" forum, but only on a copy, then when it was verified by at least several different people (with different workflows, viewpoints, methods, etc) to work without breaking stuff, *then* I'd apply the same stuff to the "live" forum.

I have no idea how practical this is...but even though it would be a bit more time-consuming initially, it would certainly save heartaches and headaches like this one. ;)
 
amberwolf said:
FWIW, stuff like that is why I would never work on the "live" forum, but only on a copy, then when it was verified by at least several different people (with different workflows, viewpoints, methods, etc) to work without breaking stuff, *then* I'd apply the same stuff to the "live" forum.

You gotta understand that making a copy of our database takes 5 hours per change you need to test for, alone. Then it requires a ~1 hour check of the site functionality at a minimum, maybe longer. I have found you can't cut corners on this time sucker by occasionally skipping the data copy process, either, due to the wide range of unexpected side effects in phpbb's code and data storage techniques.

Meanwhile, installing a plugin takes about 1-2 minutes. Pretty tempting, isn't it? Cleaning up a mess it makes takes anywhere from minutes to hours. Whereas testing takes a minimum of 5 hours, up to about 7. Testing also does not guarantee that you find all the problems. You may find them months into the future.

I'm aware this is not the ideal way to do this, but as you can see, the time economics are hugely slanted.

I have been trying to tackle the ES development problem since i took over in 2015. Of the perhaps 100 hours spent on working on ES, around 60 have been spent trying to make testing faster. I even went so far as to write something faster than Oracle's mysqldump in the process of trying to speed up database operations, which are the primary bottleneck to testing. That's how desperate i was.

I have a faster way to refresh a testing environment quickly with our next setup; it's something i discovered working with a client recently. So does Gammaray. Between these two ideas, i believe one will land. We haven't invested in this in our current setup as phpbb is not worth investing in, generally.


If you would like to volunteer to do testing, you could ensure this is done up to your standards. It is the most dull and un-gratifying job to most, so i've never asked anyone to help with it. How do you feel about doing some of this work?

amberwolf said:
I have no idea how practical this is...but even though it would be a bit more time-consuming initially, it would certainly save heartaches and headaches like this one. ;)

Actually the heartbreak to reward ratio on this one is well worth it because i will have to fix youtube embeds anyway. There's 5 different ways our system has done youtube embeds over time, and they all have to transformed before moving to a new platform. This changes my order of operation more than anything else.
 
Lol, thanks for the inspirational music.

I made a mistake in reading ES before i went to bed, and the comment from AW got under my skin, even though it wasn't meant as an attack. The frustration is more with the platform. I was up till 3am thinking of how to solve our issue with handling phpbb's data structure and the difficulties imposed by it.

There is a way to get faster disk access on amazon; you expand the size of the disk to many multiples of what you need. You back yourself into a corner of higher operating costs and the extra resources are only used 0.1% of the time. You make doing disk image backups exponentially slower, to the point of being useless. You make exporting a virtual machine for a testing environment impossible. The only way to revert this change is to have hours of downtime. It's awful. But you gain the ability to spawn a fresh testing environment in about 1.5-2 hours. When it becomes easier to test, it'll happen much more often.

I have initiated this change. It will take the entire day to benchmark the performance differences. If the extra resources become a budgetary stress, i will cover the additional cost while we are doing heavy construction. We can


The current plan for handling the youtube video issue is:

1) Transform old youtube tags into new tags universally.
2) Write some code which forces phpbb to display a random selection of posts that include
3) Write some code that translates these youtube tags back to the old style tags which xenforo can then import.
 
I have some VERY INTERESTING findings.

After installing a tune i use for another client with an wildly resource-intensive system, there was an immediate speed impact which cut the database copy time from 5 hours to 1. This increased speed took my testing time from 'all day' to hours. I was shocked.

Through progressive tuning, i have dropped this speed to 12 minutes. At the expense of high perpetual use of RAM; which is less expensive than all the extra disk i'd need.

I'm absolutely blown away at how much tuning mysql's innodb cache sizes, changing the write method to O_DIRECT, etc produces.

This REALLY big for ES. This enables me to spawn a fresh development system in about 20 minutes total. When i do that, performance of the forum is extremely poor, but stable. It also means that restoring from backup can be performed within minutes, not hours. Restoring from backup has always been a problem due to the amount of downtime it creates.

I can now craft specialized backup and restore procedures for specific aspects of the database, which would allow us to quickly recover from issues with the DB, such a plugin corrupting data/changing the storage technique.


These changes will translate to an enhanced development, maintenance, and user experience on XenForo as well.

That was absolutely worth losing sleep over :lol: :lol: :lol:
 
Back
Top