Move to XenForo software

Status
Not open for further replies.
Thank you.

The last test is being run. Fingers are crossed.
 
Fark. The testing is only halfway done. WTF :roll:
It turns out using exiftool to detect jpeg rotation puts a >3x penalty on the speed of the process. :evil:
Process completion time is more like a week, instead of 29 hours. :shock:

Time to optimize further.
 
So i had to write a poor man's profiler to get to the bottom of this.

GD ( for writing jpegs to BMPs ) and exiftool were using a majority of the cpu time... both unlikely suspects!

imagetender performance problem 2.png

I found that GD's imagebmp ( writes a bmp file ) takes 20x more time to using imagepng write than a png :lol:
Obviously GD's code for writing BMP is VERY un-optimized... improved disk speed can't make up a 20x different in performance. :lol:

I also changed exiftool to run conditionally and perform a check before copying the jpeg rotation data.. thus cut down exiftool use a lot.

imagetender performance problem 4.png

Now the load is dominated by cjpeg ( mozilla's tricked out jpeg encoder ), which is exactly what i wanted to see!

I'm amazed... the system is twice as fast, so the run through the images should be around 14 hours, not 29, and certainly not a week :lol:


Now that the performance crisis is over ( most of the high hanging fruit has been picked ), i just need to rule out that this is an answer to the rotating jpegs problem and then i should be able to move on :bolt: :es: :bolt:
 
Well it looks like imageTender finally ran both without error and without any sign of the jpeg rotation problem :bigthumb:
Looks like the final process is 18 hours instead of the original 29 as well.

imagetender final.png

Filesize checks out. Two temp files hanging out. Same number of 0 byte files. Etc.

imagetender.png

The 'adding pictures?' thread was used as a comparison for older images.
https://endless-sphere.com/forums/viewtopic.php?f=1&t=14748&start=25

ImageTender was specifically designed to not touch corrupt files. It handles the era of the most corrupt files ( 2010 ) perfectly.
I used the early posts on the stealth owners thread to go through this era.

imagetender corruption handling.png

I'd say i'm finally confident in rolling this out on the main server and then moving on to the actual Xenforo tasks now that we've massively reduced the time to xenforo import. :)


Meanwhile, mr kahuna has been working on a wiki solution, and everyone else tasked to work with php hasn't made motion. I intend to pick up the slack since those tasks are many times easier than this one.
 
It may be a few weeks until i can get started on the ES project again. But the rest is easy and the end is in sight.

Meanwhile in the office, i am setting up development infrastructure for my company as a whole so that i can hire remote workers and give them seamless access to a perfect copy of whatever server - without giving them direct access to live servers.

ES gets to hitch a ride on this infrastructure, and we will truly be enabled to make the forum a worldwide open source project. :mrgreen:

I would like to give Mr Kahuna a public shoutout for providing inspiration and wisdom recently, also making some progress on getting the ES Wiki 2.0 going already.

Ttyl.
 
Sounds superb!
great job keep it up!

thanks I really appreciate your time and effort Neps.
 
neptronix said:
I would like to give Mr Kahuna a public shoutout for providing inspiration and wisdom recently, also making some progress on getting the ES Wiki 2.0 going already.

Ttyl.

Thanks! I'm happy I found a way to give back to this community. I'm excited to see what we create.
 
The spammers just want their poetry deleted.
Whats the use? I guess if it stays their words get a higher ranking in the search engines?

The only way around it is to have a waiting period when new users join, and to force them into a Welcome Thread where they have to post before moving on to posting in other threads.
I believe Nep talked about that and thats been squashed, but you are right its not really that bad, I just find theres a little more of it lately.

Altenwerth said:
They are fierce warriors whose movements and speech provide an amusing display of double trouble.



ZeroEm said:
Thought it was better the last few weeks?

by markz » Oct 11 2021 12:18am

Why is there more spammers these days?
It's probably them going through all the anti-spam security just to end up with 1-4 posts before getting the hammer.
 
Many phpbb anti spam plugins no longer work.. and spammers come in waves.
Our anti spam systems are simplistic and other options are not effective or feasible for moderator time.

..another reason why we're moving! Xenforo's got the anti spam features nailed down pretty good! i don't want our moderators wasting much time with that, nor the poetry being read for long. :)

PS
Thanks for the thanks, everyone.
I'm trying to clear out a good chunk of my mid-priority stuff so i can throw some serious time at ES. :thumb:
 
Is there a quick links in the new XenForo, in phpBB there is a quick links next to FAQ near the top header, have an additional "Your Threads" along with "Your Posts"
 
That's a bare minimum feature and if it doesn't have it, i'll hack it in :)
 
ImageTender is being deployed today. Expect a little mayhem, but do know that i have backups. :)
 
OK.. there is a HUGE rift between the speed of AWS versus the 10th gen core i7 i tested the process on. About 3 times slower.

ESCompressor.png

The server will be in and out of consciousness in about 10 mins as i give it a little more zip.
 
This is unusual, but the code is performing as expected without bugs and doesn't need to be babysat. :lol:

I think you are not going to see any hiccups for the rest of the day while this thing is savagely beating on the CPU.
I overestimated the process time and think we'll have results by tomorrow.

In the meantime, you may end up seeing an image in a half-processed state.
 

Attachments

  • imagecompressor.png
    imagecompressor.png
    27.8 KB · Views: 90
neptronix said:
OK.. there is a HUGE rift between the speed of AWS versus the 10th gen core i7 i tested the process on. About 3 times slower.

ESCompressor.png

The server will be in and out of consciousness in about 10 mins as i give it a little more zip.
You mean those AWS CPUs that Amazon made them selves for their AWS services?
Cool, yeah I read about those in tech news somewhere, it looks like just about every tech giant makes their own CPUS now if it doesn't require x86 binaries.
In this article they are called Graviton2, you can even see "an Amazon company" on the CPU in small writing.
https://www.anandtech.com/show/15578/cloud-clash-amazon-graviton2-arm-against-intel-and-amd
car_678x452.jpg


Looks like processors are turning into a easily available commodity from a lot of sources instead of just something you could get from AMD/Intel/Sun like the old days.
 
I think ARM processors are pretty cool, good performance per dollar, but haven't been produced with great clock speeds just yet.. many important time sucking processes on a site like ours are single threaded, so a small amount of very fast cores is needed... we stick with x86 for the time being.

Also, i'd like a precompiled binary package for everything and i'm not sure if i get that with an ARM version of a linux server distro. Haven't looked into that though.
 
Done. Freed up about 29gb of image pork. The disk was about to be full before. :bolt:

done.png

So glad it's done!! now.. to set up a fresh new dev system for the remaining work and testing some of the other prerequisite surgeries.
 
Okay, repairing the corrupt image files from the past and will then upload them back to the server.
After that, i'm refreshing the development machine for the volunteers to use.

I'm also setting up my first micro data center. Got a fixed IP out to the world and a secure location.
The ES Dev system hitches a ride for free.

We do have some cool shit planned and i'm sorry it's taken a while!
 
Things are going well. I still have some things to clear off my plate before i get heavy into another ES round though.


Because the crypto miners had to buy full computers to get GPUs for a while, there's shit-tons of spare GPU-less and new machines being hawked on ebay. I bought ES + my company a mini Dell server with a i7-10700 for $460! and 1 year old!

It replaces it's 'baby bro' to the left that only had a core i5.
( baby bro was the dev box until i found out imageTender would take >40 hours running on it; i had to run imageTender on my own 10700 during testing!! and that was nowhere near fast enough! ).

img.jpg

It will be a great dev box for the $ and we need that 4.8ghz peak for crunching gigabytes of data for hours on end.

Should have it set up fairly soon and unleash the Kahuna on it to have him set up the new wiki ahead of the xenforo move.
 
Just curious: Harddisk, or SSD type? I've seen that SSDs seem to be way way faster for random access....


Also, unrelated, but is it possible to "move" a complete harddisk clone from an old Win10 laptop to a new drive on a different machine (like one of those you've got there), without having to reinstall Windows (just reactivate it or whatever) so that I don't have to reinstall and reconfigure a bajillion pieces of software, music plugins, etc? There arent' any install media, just a restore partition, which presumably only works on the laptop....

I ask because my laptop isn't quite sufficient anymore for what I want to do with music, and someday when I have a job again and enough money coming in I'd like to use a desktop for the music stuff for the expansion capabilities and multimonitor possiblities. (and the laptop is having more and more random issues with hardware and/or crashes over time).
 
Status
Not open for further replies.
Back
Top