Move to XenForo software

Status
Not open for further replies.
amberwolf said:
Just curious: Harddisk, or SSD type? I've seen that SSDs seem to be way way faster for random access....

I have a 1TB WD Black NVME drive. This thing is no joke. Modern mechanical hard drives have horrible random access and maximum thoroughputs of 400mb/sec ( and we're talking about 10K server hard drives here, consumer grade stuff's usually in the 100mb/sec range )

wd black.png

I bought this to help reduce the time we're going to take to do manipulations of LARGE sets of data, which is a MASSIVE pain point for system administration and web development of this system.

As for your situation, you could buy the cheapest NVME drive in the world and still blow the doors off any mechanical hard disk - you will see pretty much any computer get multitudes faster by switching to a NVME drive. A SSD that connects to the serial interface like a 2.5" SSD is unfortunately not going to be as big of an upgrade but on an older computer, it's going to be a HUGE upgrade.

amberwolf said:
Also, unrelated, but is it possible to "move" a complete harddisk clone from an old Win10 laptop to a new drive on a different machine (like one of those you've got there), without having to reinstall Windows (just reactivate it or whatever) so that I don't have to reinstall and reconfigure a bajillion pieces of software, music plugins, etc? There arent' any install media, just a restore partition, which presumably only works on the laptop....

I ask because my laptop isn't quite sufficient anymore for what I want to do with music, and someday when I have a job again and enough money coming in I'd like to use a desktop for the music stuff for the expansion capabilities and multimonitor possiblities. (and the laptop is having more and more random issues with hardware and/or crashes over time).

With Windows 10, you can often move a hard drive, and this is because they changed the windows license authentication model. Give it a try.

I have that problem you have, and these special software setups ( like my music production setup that has ~80 plugins all finely tuned ) all got turned into virtual machines a decade ago. Thankfully, virtual machine technology has progressed to the point where you can pass a GPU card to them ( Linux only ) and also directly pass a USB audio device with very little latency ( Virtualbox, etc ).

Only in the last few years has it been possible to get good, low level, runs all-the-things virtual machines.

In fact, i'm typing to you on my work machine, which is a windows 7 virtual machine with dozens of preconfigured, expensive, and elbaorate tools for systems administration and web development. Absolutely no ****ing way would i want to ever set this up again, so i've enshrined it in a VM, and that VM 'hard disk' sees regular backups itself. I can also take that VM hard disk and put it on a thumb drive, move it to another computer, and have that machine booted and running in a minute.

If you are a power user of computers, i highly recommend you think about moving things to a VM.
 
nicobie said:
I think Gamers Nexus is #1 when it comes to building computers. I like his no BS attitude. Hardware Unboxed is pretty damn good too. Linus kind of gets on my nerves. Paul's Hardware is useful and Bitwit is funny, although he was better before he split with his "wifey sauce".

Gamer's Nexus is quite decent. My own computer designs usually involve finding breakneck efficiency though, so i don't pay too much attention to gaming outlets because their focus is raw power.

Nobody really covers the high efficiency stuff!!
 
ActuallyHardcoreOverclocking
https://www.youtube.com/c/ActuallyHardcoreOverclocking

Is pretty good and what I go to for mobo's and ram info.

I haven't found anything else but I haven't looked either because I am not looking for a new computer. Every now and then I just like to surf and see whats out there and play with pcpartpicker website.
 
The new development box is now setup and open to the internet but a secret to anyone other than the dev team.

I did a roll call and so far mr kahuna is on board with the wiki project but none of the other programmers raised their hands yet.
I get the feeling i'll be doing all the coding for the remaining tasks but at least we have a very very fast box to crunch data with and the remaining tasks are also magnitudes easier.

Admittedly it did take me a while to engineer a very good 'dev shop' process and set of protocols for safely handling our site over to members here and i think i may have taken some steam out of the volunteers.

But the end goal is to make ES easy to hack on for people who are willing to throw code at making it better, because we are thinking many years into the future here. :bolt:


Well, business is slow here and i am catching up on the huge lowish priority queue, so i am kind of plotting an ideal time to go hogwild on ES coding.
 
Honestly, being part of the moderator team is enough. Especially if you stick with us and continue doing that very important work.

Me and Justin both have some pretty big visions for what ES could be and want to put a really big investment into hopefully sticking around for another decade or two. Having people manage the social environment while we work on making ES technologically awesome is invaluable. It's nice to wear (mostly) one hat here.

I'm very thankful for the moderators sticking with us for so long!!
 
The dev server is alive and kicking and running a censored copy of our exact setup.

img (1).jpg

So glad this is done and we can move on to greater things. :)
 
Extremely good news today..

2021-12-15 12_22_36-Finalize import _ Endless Sphere - Admin control panel.png

Note: this is the default xenforo look, not the final one. ---v

2021-12-15 12_49_34-Endless Sphere.png

Between the image processor, some pretty mondo hardware, and some additional server performance optimization, the xenforo conversion time got cut from 8 hours to about 3 hours.

This extra speed allows us to write the content fixing processes so that they run after the import... and will probably give us a total downtime of just 4 hours.

I have a call with Justin on friday to show him the system and help make some design decisions.
I also have a decent amount of time to work on this because business has been a little slow lately.
It's happening!
 
Gracias.
It's nice until you start clicking on things and notice which parts of the conversion process are broken. :lol:

At least the post editor is as promised: awesome.



When we get closer to version 1.0, the forum will get a more formal, hands on preview.

Now it's time to write some conversion code. I'm going to start with the hardest one - the youtube tag problem.
 

Attachments

  • 2021-12-15 18_10_25-File attachments pruned - let me know if you see missing images. _ Endless...png
    2021-12-15 18_10_25-File attachments pruned - let me know if you see missing images. _ Endless...png
    137.1 KB · Views: 53
Side note.. xenforo's database is many times faster than i expected. This pull of all posts with the youtube tag in them is like 100x faster than the database pull out of phpbb. I don't know why. This used to be in the minutes range, not seconds.

2021-12-15 18_56_56-https___dev.endless-sphere.com_admintools_endless-tools_phpbbYoutubeFix.png

So these conversion tools should be way easier to write than i expected since the testing time just got cut down to a fraction :bigthumb:
 
I have some even better news.

Xenforo converts the difficult to parse and properly interpret 'new' style youtube tag to the old style.
The one thing it fails to do is convert the youtube embeds to it's own format.
Funny behavior... but i'll take it.. because that last step is much easier.

A bit of code written in a sprint last night is up to about 95% accuracy.. it is a few adjustments and some testing away from being done.

I'm going to go on a big coding sprint and see if i can also write the remaining conversions today.
Wish me luck.
 
The youtube fix is in!!

My interpreter: recognizes all but 166 youtube tags.
Phpbb: recognizes all but 817 tags in this test, but fares way worse in real life because it doesn't recognize many tags prior to ~2020.

Of the youtube tags my system can't recognize, about 99% of them are severe user mis-types:

2021-12-16 16_23_16-https___dev.endless-sphere.com_admintools_endless-tools_phpbbYoutubeFix.png

The real problem was how many variants of how youtube tags are typed.

2021-12-16 16_36_50-https___dev.endless-sphere.com_admintools_endless-tools_phpbbYoutubeFix.png

The working solution - for anyone interested:

Code:
<?php
//fix youtube tags in phpbb/xenforo
//new phpbb tag: <YOUTUBE content="sn19xvfoXvk"><s>[youtube]</s>sn19xvfoXvk<e>[/youtube]</e></YOUTUBE>
//old phpbb tag: [youtube][/youtube]
//target: [MEDIA=youtube]HjxZYiTpU3k[/MEDIA]

db::switchDB(2); //use xenforo database

//get array
$postData = db::getArray("SELECT post_id, username, FROM_UNIXTIME(post_date), message FROM xf_post WHERE message LIKE '%youtube%' OR message LIKE '%YOUTUBE content=%'");

zl::quip("Records:" . count($postData));
//html::printTable($origWithUrl);

//accuracy counting
$comprende = 0; $noComprende = 0;

//strips everything to the right of a character, including the character itself.
function stripRightOf($inString, $truncateString)
{
	if(strpos($inString, $truncateString) !== false ) { return substr($inString, 0, strpos($inString, $truncateString)); }
	else { return $inString; }
}

//returns an array of text that is between tags.
function getBetweenTags($inString, $startTag, $endTag, $includeTags = false)
{
    $result = array();
	$chunks = explode($startTag, $inString);
    foreach($chunks as $chunk)
	{
        $pos = strpos($chunk, $endTag);
        if($pos !== false)
		{
			if($includeTags) { $result[] = $startTag . substr($chunk, 0, $pos) . $endTag; }
			else { $result[] = substr($chunk, 0, $pos); }
		}
    }
    return $result;
}

//chunks to delete ( usually the front )
$replaceArray = array
(
	"[youtube][Youtube]",
	"[/youtube]",
	"[youtube]",
	"[youtube",
	"[yotube]",
	"https://www.youtube.com/edit?o=U&video_id=",
	"http://www.youtube.com/watch?time_continue=1&v=",
	"https://www.youtube.com/watch?time_continue=1&v=",
	"http://www.youtube.com/watch?feature=player_detailpage&v=",
	"https://www.youtube.com/watch?feature=player_detailpage&v=",
	"https://www.youtube.com/watch?feature=player_embedded&v=",
	"http://m.youtube.com/?client=mv-google&rdm=4owzg9jvi#/watch?v=",
	"https://www.youtube.com/embed/",
	"http://www.youtube.com/embed/",
	"http://www.youtube.com/v/",
	"https://www.youtube.com/v/",
	"http://www.youtube.com/watch?v=",
	"https://www.youtube.com/watch?v=",
	"https://youtube.com/watch?v=",
	"http://youtube.com/watch?v=",
	"http://m.youtube.com/watch?v=",
	"https://m.youtube.com/watch?v=",
	"https://youtu.be/",
	"http://youtu.be/",
	"www.youtube.com/watch?v=",
	"youtube.com/watch?v=",
	"player_detailpage&v=",
	"youtu.be/",
	"watch?v=",
	"https://",
	"http://",
	"[url]",
	"[/url]",
	"?v=",
	"v="
);

?><table class="zl_table">
	<tr><th>old</th><th>new</th></tr>
<?php
foreach($postData as $post)
{
	$replaceStrings = array(); //what strings to replace?
	
	//pick apart youtube tags
	$tags = getBetweenTags(trim($post['message']), "[youtube]" , "[/youtube]", true);
	
	if(!is_empty($tags))
	{
		foreach($tags as $tag)
		{
			$newTag = str_replace($replaceArray, "", trim($tag)); //strip what we can
			
			//second phase strip after tag
			$newTag = stripRightOf($newTag, "&");
			$newTag = stripRightOf($newTag, "?");
			$newTag = stripRightOf($newTag, "#");
			$newTag = trim($newTag);
			
			//meets valid criteria or nah?
			if(strlen($newTag) >= 9 && strlen($newTag) <= 12)
			{
				$newTag = "[MEDIA=youtube]" . $newTag . "[/MEDIA]";
				$replaceStrings[] = array('old' => $tag , 'new' => $newTag);
				$comprende++;
			}
			else
			{
				//$replaceStrings[] = array('old' => strings::htmlToText($tag), 'new' => "<font color=red>" . strings::htmlToText($newTag) . "</font>");
				$noComprende++;
			}
		}
		
		//replace and update the post in the database.
		if(count($replaceStrings) > 0)
		{
			foreach($replaceStrings as $replaceString)
			{
				echo "<tr><td>" . $replaceString['old'] . "</td><td>" . $replaceString['new'] . "</td></tr>\n";
				$post['message'] = str_replace($replaceString['old'], $replaceString['new'], $post['message']);
			}
			
			db::writeRow("UPDATE", "xf_post", array("message" => $post['message']), array("post_id" => $post['post_id']));
		}
	}
}
?></table><?php
echo "<br>Comprende: " . $comprende . "<br>";
echo "No Comprende: " . $noComprende . "<br><br><br><br><br><br><br><br><br><br>";
?>
 
Well my brain's shot. But i also got a hell of a lot done. :bolt:

Youtube import + tag fix is tested and working.
Banlist and permissions import is done.
Other things i decided to be a manual process not worth automating.

One glitch that will bug everyone: everyone shows up as a new user from default until they log in, only then do they get their previous title of 'active member'. Attempted to write a script to do this, but there is some kind of restriction on updating the xf_post table that i don't understand, and have never seen before.

I can work on a fix for that after the fact and have two plans of attack for that. But it's basically just an irritance.


Remaining steps:
Talk to Justin about design decisions that affect the use of the platform.
Install and test plugins.
Open the developer site for moderator and user review for feedback.
Adjust and then set a deployment date.

The rest is easy! we're close! :shock:
 
The real problem is that, to the world, our forum would appear broken, because it should not consider someone with a 15 year history to be a 'new member'.


I can't wait either, but now i'm feeling a nervousness about changing the site so dramatically. I am currently working on the Xenforo theme redesign so that it resembles the display density and colors of ES so that it's much less of a shock to everyone's eyeballs when we flip the switch. Also, some parts of Xenforo's text are borderline unreadable on high resolution displays like the 27" 1440p display i have, so that needs work.

By the way, i have noticed that Xenforo is essentially a commercial version of phpbb, or at least, they copied many design patterns and have feature parity in weird places and have a few identical database structures ( field names and all ) with phpbb. So i'm thinking that xenforo's behavior as a whole will not come as a shock at all, and people will see it as a much enhanced phpbb experience as i do.

We'll see!!
 
nicobie said:
I don't think it's too much to ask that members will have to log in one time after the switch. I think some forums do that on purpose every once in a while to try to slow the spam.

Well done. Can't wait. 🍀

Some older members might log in every few years. Or never.
 
Starting to working on that familiarity, contrast, and display density adjustment.
Stock configuration on the right, Neppy's configuration ( so far ) on the left.

2021-12-17 13_45_33-Endless Sphere.png
 
neptronix said:
One glitch that will bug everyone: everyone shows up as a new user from default until they log in, only then do they get their previous title of 'active member'.

That's actually fine, since it will show who is actually active nowadays. :)
 
Speaking of too big of screen problems.. the backend has it pretty bad.. in addition to not having a width limiter.. and lots of very low contrast fonts.

2021-12-17 15_01_33-.png

It will need a beating too.
 
Status
Not open for further replies.
Back
Top