Searching For Text/data Extractor..

..pull out copies of my posts for the last 3.7 years. I belong to a forum, that has 5,059 posts of my own, on 196 pages. I tried UI-Vision, Instant Data Scraper, WebScraper Pro, and at least 1 other I can't think of right now.

I'm only trying to get my own posts, and can copy a full page in "My Private Post" list on the site. I was also offered a PII Data File (xml) which was not my actual posts. I only want something that someone has used for themself, as I have googled several things I can download and try. I'm tired, and want to move on with this project.

I figured I could copy and paste it all, if I do 10 pages a day, I could have it all in 20 days if I don't go blind or wear out my keyboard.

It's pretty nicely laid out in the "all posts made by me" section, just 2 columns, on left is the menu column.

Can anyone suggest something I can try next? And maybe even something that will "select the whole column" Option to "continue to next page" and be able to d0 all 196 pgs, then export a file, CSV I think for note pad?

I have Windows 10 on my desktop, but Laptop with Win 11. Firefox Browser, and my website has Cloudfare so I think the first thing I tried was blocked. I don't want anything Cloudfare would block. Any help will be appreciated Suggestions, ideas more than welcome as well!

 

Possibly the best option is a userscript that can be run using Home | Tampermonkey. The userscript would run inside your browser (Firefox) so it does not need to bypass Cloudflare or send your data elsewhere. A userscript could visit the nearly 200 pages slowly, collect only the post column and then download as a CSV.

You would need some detail about how the posts are coded in a web-page. For example, open one page inFirefox, right-click the text of one of your posts and choose inspect. In the Inspector, look for a repeated element surrounding each complete post, something like ".post", ".message", ".postbody" or similar. Then some information about how to traverse from one page to the next, again right click inspect the next button or similar. Using this information, build up a user script to scrape pages with a delay in between each next page to avoid lock-out due to too many requests or triggering rate limits.

 

This sounds very good!

I'll go get the Tampermonkey, and look it over. I want to thank you so much for sharing your info with me. I'll come back, of course, and let you know if I think I can understand how to use it. Also, I'll look at the details of a post and let you know about that. Seems like it was Javascript but that came from AI, so it will be good to learn for myself

 

I think that I want to try the Tampermonkey, but to be honest with you, not waste your time Maxwell, I asked AI to help me do the script. It is Html, I found out that much but overwhelmed me thinking I could do it on my own. I don't want to take advantage of someone willing to help me. So, I have a script now, made by AI but would like your input on whether or not it looks "good to go" to you?

The script reveals some things but I am writing a book that I hope many people will see so I'm not worried about that

 

I do like the Menus in Tampermonkey, best I've seen so far. I can follow them pretty well so far! I went ahead and tested the first script, failed the "next button" part, AI gave me a new one, failed again. Then I found out it didn't change the script, LOL, at all, oh dear

I don't want to go round and round with AI again, so your help in the script part that controls, my first page, moving on to the remaining 195 pages, would help so much.

When I try to highlite (to inspect) the Next button on my website, it I can't get it to highlite, just very light gray, but I'll show you a snip of it's info I do have. I feel like my brain is scrambling again, so much I don't know but hope this helps..I didn't attach it and should have..

 

Your script has an error in it for selecting the next page. It should be two underscores rather than one. That is, should be:

const nextButton = document.querySelector(
'li.ipsPagination__next a[rel="next"]'
);​

 

Thank you, got that fixed. Something else was wrong at the end of script, got a red flag.
I'll put a snip, AI says something about it has to match the change above, underscore. I'm so tired again so I have to stop and rest, but I'll get you the area that has to be fixed, according to AI anyway. Forgive me I'm just dingy from working on this since 6 a.m.
I've run the test page (page 1) had to fix "next page" code, and the two things attached. Also, even though I do a save file, I'm not seeing a .txt file on my PC. AI said it should be there, and we set the file output as .txt.
I'll be back up in a couple of hours, but no hurry on your part.
I was glad to see your input for a definite human- 2nd opinion, thank you again Maxwell

 

I LOVE THIS TOOL @Maxwell, "TamperMonkey", freaking amazing!! It is working beautifully after 5 days of struggling to find the right "tool". Wow, if I can help you back Maxwell, or pay it forward on this forum I will be doing just that!!

First, clean copy of Page 1, awesome, and I can put back in the "next" page code later when I am rested. Took that out for the 196 total pages to do the 1 page test. I'm sure it will work, maybe some tweaking but will do!

 

OK, I don't think adding run is going to help here.

One issue is that the script calls URL.revokeObjectURL() immediately after a.click(). The click itself is synchronous, but the browser processes the resulting download asynchronously. Revoking the temporary file URL immediately could happen before the browser has started using it, so a short delay is needed.

The script also creates a temporary download link. Although an unattached link can often be clicked successfully, attaching it to the document before clicking it makes the behavior more reliable across browsers. The link is removed immediately afterward.

The file should normally appear in the browser’s configured Downloads folder.

I updated the original script with comments and log messages. I also renamed some variables to make their purpose clearer - see attached text file. Also, added a function to ensure page rendering by any JavaScript is complete, but this may be optional if the posts are already present in the page when the userscript starts.

 

Run your own LLM on a USB SSD. It'll be uncensored and without limits. Ask it to scan for your name/posts and copy them into whatever format you like.
I tried it with Google Gemini and it says it "can't" do it. With your own LLM, you won't have this limitation.

https://github.com/Lingdas1/local-llm-guide/blob/main/blog/welcome-local-llm-guide.md

 

I'll keep the link, for just backup right now @xrobwx71 . I think I am almost there with the TamperMonkey. It's very user-friendly, better than the 3 or 4 things I've tried so I want to stick with it, I thank you for your feedback I a will let you know here if I decide to use it at a later time

Just an update here for you as well @Maxwell. I'm not giving up yet, as it went so very well today. I am way better at working with AI, and I got as far as passing the test for copying one page of posts, 25 or them. Amazingly organized in a .txt file! Also put in a code for "next page" but it failed, only ran it for 3 pages so glad I didn't go for the 196.

I thought I'd put up that last copy I have with what was changed, and tested. Everything you suggested was used, as it matched what AI was giving as well for script. I can do that part, have no idea how to write the code of course. It's so fun and interesting, I stay with it longer than I should.

So I'll attach a new copy of script, I so appreciate your help of course, and if you get the chance, maybe you'll see where the next mistake in code is. No rush, I need to slow down a bit anyway..File attached below:

 

This latest script is missing a brace to close the run function before it executes the run function in the outer anomymous function. It should be (see attached corrected code):

Code:

            } // closes the else block
        }     // closes function run
        run(); // invokes the inner function
    }         // closes the outer anonymous function

)();

​

The script only traverses 3 pages due to the 3 on line 39:

if (nextButton && nextButton.href && Number(nextButton.dataset.page) <= 3) {​

 

Use Online JavaScript beautifier to format the code, so that matched indentation levels become more obvious and various code constructs are color coded..

 

What a wonderful sight to see on wake-up Maxwell!!

I'll be able to finish my initial setup (get my copies, start editing) with your help today! I don't know what I'm more excited about, but the way TamperMonkey lays out the copy is so amazing. I also like the sound of the Online Javascript beautifier! Also, the fact you understand exactly what I try to convey in my "Help Me" posts/replies Not knowing the right questions is my biggest issues, thank you again

 

I just realized this 196 pages comes out in a single, text file. Kind of freaked me out to try and wrap my head around that. Maybe needs a code change that would separate in some way?

The other thing is, I don't know if I can even open it when it's done running, which is just another few minutes. So some other software that might be known for editing a big amount of text.

Any info, ideas, still very welcome. So much I don't know about what I am attempting to do but I never thought I'd get this far Job just now finished, I'll find out more now, when I try to open it. If my PC starts smoking, I'll go take a break

Edit: I think I should have coded it to make smaller files. Feedback still welcome on that.

 

Ok, file didn't have a chance to save because I stopped the TM from running. It kept running because I didn't have the file ready to save to my pc, TamperMonkey causes my File Explorer to open and save new file. That hadn't happened so I took the chance of stopping it.

So now, I'll be searching to see how to code for separate amounts of text files. Since I have 196 pages, dividing those up to sizes that Notepad can handle. Not sure the number of files yet.

Can you help me Maxwell, with the coding changes I need to just run say 25 pages? I am thinking all I need to do is change the numbers in this code attached. Then I am guessing only, but just F5, to run TM as many times as it takes to get all 196 pages, but in separate txt files, is this right?

Not sure how to change the code, but I am thinking 1-25, then next change to 26-5o etc?

 

I got it!! Running smaller batches, although I am changing it for each new batch manually! I don't mind, as I do understand TamperMonkey has a "batch" option or Java has a code for it. Just figure this is very good "enough" for now! Easy code change, just the next "ending page number" with the <= 25 (then 50, 75 etc.)

Replies welcome, so appreciate all the help I got here! I'll make a new topic when I may need some suggestions for Software that does Editing!!

 

nevermind, I just can't use AI any longer for a project like this. I showed it screenshots of my code changes, and it agreed it right. It was not. Only the ones you gave me @Maxwell have worked.

If you will help with coding changes to run the batches, I will use "them only". I know I sound stressed, and it's because AI seems the easiest fastest way, and I was wrong. What's good is I have learned a lot about TamperMonkey, and I know I can do this, with your help. Attaching latest code/script for you, whenever you might be able to help me a bit more:

 

forgot attachment, apologies:

Area I was changing the page numbers for batch, left the rest of code except number for last page. AI said it all looked good, but it doesn't work. I'll explain more if you need me to gladly,

 

I'm sorry about all the posts, but now I am no longer working with AI, and feel I need to find a program that can search javascript to find errors in my code. I looked at the Beautifier, but I don't see where it might help with finding errors, any help appreciated?

I'll come back an update if I find something

 

Quite a few things to answer!

1. The Online JavaScript Beautifier only tidies any pasted code when you press the Beautify Code button at the top-right. I used this as a very simple check to verify the structure of your code. It can make mismatched braces easier to see, but it does not prove that the code works and usually does not identify logical errors.

2. Changing the limit number from 3 to 100 or 196 causes the script to traverse all the pages from the first page to the limit specified. It does not output selections or sections of pages, just all the pages from the first to the limit number. You might just end up with a text file with duplicate pages if it accumulates the pages with each run into the same file.

3. For developing JavaScript code, there are numerous tools listed here: Download Programming Tools for Windows - MajorGeeks . Two good ones to choose are Download Visual Studio Code 1.136.0 - MajorGeeks and Download WebStorm 2026.2.0.1 - MajorGeeks - but these will take time and effort to understand their use. There are probably many YouTube tutorials on the subject.

4. For editing text files, there are numerous tools listed here: Download WordPad and NotePad Alternatives Tools for Windows - MajorGeeks . A top general purpose one to choose would be Download Notepad++ 8.9.8 - MajorGeeks which can also help with viewing and formatting JavaScript (but it is not a development tool). If you want a Microsoft Office like tool for finalising the text into a document, then Download LibreOffice 26.8.0 - MajorGeeks is a good alternative.

5. Now, the final challenge to modify the script to output batches of 25 pages (more or less) at a time to separate files. Before modifying the script to create separate batches, it would be useful to determine the size of the complete 196-page result. If the file opens comfortably in Notepad++, it may be simpler to make one complete extraction and then split and curate the text manually. If it is too large, slow to open, or contains substantial duplication, modifying the script to create separate 25-page files may be preferable or a script to process the 196-page resultant text file into smaller files.

 

Yes, I got carried away with my posts, and you've gone above and beyond to help me. Your feedback is refreshing though, thank you much!

The thing that I can't agree with more, is to learn as much as I can on my own. Especially on coding, but honestly wondering how much of that I will need in the future. It's fascinating, and if I stay fascinated with it that will keep me wanting to learn. I'm 73 now, learning things I was never drawn to. I'm surprised at what I am able to learn, plus the fact, most seniors, offline I meet, don't want to learn anything new.

I understand about the "Beautifier" now, and learning more about coding first seems the best approach. I actually think, for now, going ahead with the 196, one document approach. I didn't even try to open it in Notepad, and hadn't figured out the total size. I insisted on stopping it from running. I guess I was afraid my PC would start smoking

Thank you for your link suggestions for developing Java Code, and editing, I will take a look today. Especially the editing as I do have Notepad that comes on Win 10 to at least "try" first on 5059 post/196 pages

This above is what my focus is on today, seems the best "first thing first", seeing if it opens for me.

Thank you again Maxwell, love the avatar as I still go out of my way to find Maxwell House coffee

 

My TM looks like it completed the 196 pages (I see on the tab for my website, 196. It blinked at each new page, changed to the next, normal. But when it reaches 196, it keeps blinking and hasn't "saved" yet to my File Explorer. This probably takes time, not just to copy of course, but made into the .txt file, I'm assuming only?

I did estimate 945.7 MB/figure 1 GB of data.

I'll see how long it goes before saving, then may have to google to get an answer, have not used AI at all today, done this all on my own, re-reading all you've shared with me @Maxwell, way more confident with that.

 

another horrifying thought, is this taking TM so long to process because it's actually going all the way back to page 1, on each new page? I have to re-read everything to be sure I am getting it, but if this is true, I won't stop the "run" anyway, see if I can open it. I didn't do the Math right, so it's normally "scraped" for "just each" page and that's 320 MB. No idea if it is adding in page 1-whatever

I did rely on AI yesterday and it said the <= plus the "last page" number was correct, I sware if I never use AI again, I'll be shocked that I didn't learn my lesson

 

Last one, I asked AI again, I must need to go to "AI Anonymous" now!

I did misunderstand # 2 of your explanation. I understand now that the slowdown isn't from creating the final file, it's from processing and storing all the accumulated data as the scraper goes through the pages.”

Here is the new script "it" says I can run to stop the long processing TM is doing right now., still. I won't stop it yet, but I'm thinking it may be easier to just bag-it, and start over with script. I'm not going to start over though, until I get to hear from you, and that's a promise to myself

 

The script collects the posts from each processed page into Tampermonkey’s persistent storage, rather than appending directly to a disk file. It retrieves the previously stored array, adds any new posts from the current page, saves the updated array, and then proceeds to the next page. Once it reaches the stopping point — normally when there is no suitable next page or the next page exceeds the specified limit — it converts the accumulated data into one text file and initiates the download. Tampermonkey storage will normally persist across browser closures, browser restarts, page reloads, and script restarts.

If the script is run with a lower limit and later re-run with a higher limit, the second run will reuse the data saved by the first run. Without duplicate checking, the second run could store the earlier pages again and produce duplicated results. However, this statement:

Code:

if (text && !saved.some(p => p.text === text))

checks whether an identical post text is already stored. If it is, the post is not added again. Therefore, unchanged posts from the first run should not be duplicated in the second run’s output. The original stored entries remain in Tampermonkey storage, but only one copy is included in the downloaded file. My earlier concern is mitigated by the code statement.

When processing finishes, the script creates a new download named:

Code:

benzobuddies_posts.txt

If that filename already exists, the browser will normally rename the subsequent download to something like:

Code:

benzobuddies_posts (1).txt
benzobuddies_posts (2).txt

This filename change is controlled by the browser, not by the script. Depending on the browser’s download settings, it could instead prompt about the existing file or offer to replace it. The script itself does not open or append to the previous downloaded file.

Regarding the timing, processing pages 1–196 requires approximately 195 ten-second delays, or about 32½ minutes, plus page-load and storage overhead. Reaching page 196 shows that the navigation phase has completed, but creating the final large file may still take additional time. The prolonged blinking is more likely caused by processing the accumulated data and generating the file than by the script repeatedly going back through earlier pages.

Overall, if the first script reached page 196 and completed its download, the resulting file should contain the entries collected while processing pages 1–196, with duplicate entries omitted when their extracted text was exactly identical. If data from an earlier run remains in Tampermonkey storage, the file may also include entries retained from that earlier run.

There is probably no need to run a replacement script if the first script completed successfully. The replacement script should not be run without correction: its post selector uses "div.ipsEntry_post" rather than the original "div.ipsEntry__post" (double underscore), it has no page limit, it revokes the download URL immediately, and it clears the stored data after initiating the download.

Once you have the final text file, it may be better to split it into smaller, more manageable chunks if it takes too long or uses too many resources to open in Notepad++, rather than modify and run the scraper again.

 

I did have to get some help from AI to understand better, what you took such a lot of time to provide for me. If I had it figured, along with AI's help, I just ran the same code again, but I'm doing it overnight, or at least 3 hours it can run before I hit the sack. I'm not staying on here, so just want to thank you for all your help, again!!

I guess the truth is, I needed you (my human help, and AI). I will let you know how it goes, update the thread. I have a good feeling it will do the job, I just stopped it too soon 2 times now I thank you for all I've learned from you @Maxwell

 

Well, it's run since 4 pm until 3:41 a.m. and still no sign of it finishing. One thing I noticed last night just before bed, was the appearance right next to the Firefox tabs, one for the TM and one for my BB site I'll put below.

I'm not sure what to do now, since I don't want to lose what might be "saved" somewhere. As always, appreciate any input. I never touched anything after I started the run, except turning off the monitor before 7 when I hit the sack

 

Given that the script appears to have been running for an excessive amount of time, I would do the following:

1. Check the Downloads folder and Firefox’s download history for evidence of files containing the extracted data.
2. Check Windows Task Manager for Firefox’s CPU, RAM, and disk usage. This may indicate whether the script is still actively running.
3. If CPU and disk activity are nearly idle and no download is appearing, assume that the script has stalled.

4. First stop or disable the userscript if possible, without closing the forum tab or Firefox.
5. Inspect Tampermonkey’s stored data for saved results from this script. Open the Tampermonkey Dashboard, click the script’s name, and select Storage, if available: FAQ | Tampermonkey
6. Only after checking or copying any stored results should you close the forum tab, if necessary.
7. Do not rerun the script until it has been established whether useful data was saved.

The stored data may survive stopping the script if the userscript has been saving checkpoints through Tampermonkey’s persistent storage. The missing final download does not necessarily mean that all collected posts have been lost. The script may have saved the accumulated posts but stalled while writing them or creating the final downloadable text file. If the posts existed only in a temporary JavaScript variable, however, closing the tab will lose them. Any data from the current page that had not yet been saved would also be lost.

This increasingly suggests that the script should generate output in batches as it traverses the forum — for example, after every 10, 20, 25, or 50 pages — rather than accumulating all the processed data in memory and creating one very large output file at the end. It would also be preferable to save only the actual post text, rather than complete page or post HTML, and to clear each completed batch after it has been downloaded.

 

Storage is not present Maxwell. I did stop it running as I just don't see anything at all that shows it ran, except for on the Tab for my site, it says 196, so something completed, even if it was just using it's fingers.

I got hold of a new script from AI if you want to check it for me. I'll put it below, along with the info from AI. Only a couple of notes from AI, short notes.

So the .txt file has my notes in there, not bottom-line script to paste. I titled it wrong, it should say "from" not "for Google AI":

 

It's all done, thanks to your help Maxwell. I'm exhausted, so ready to move on to something elseEnded up with all of my posts. They are in one document, but it opened fine in my notepad. I will look through the editors here on Majorgeeks, it's the only place I am comfy downloading. I do want to pick one that has a few more bells and whistles than my Notepad, but we'll see.

The TamperMonkey worked so well, and you had a huge amount to due with my success in "rescuing" a copy of my writings The final Script ran well 42 minutes. Except it continued trying to go to the next page/post when it reached the last post. The little status message in TM finally showed up (in the last script edit) plus a link to save anytime during scrape so I knew I'd get all my pages this time.

I'd be glad to put the final script here if you would like to have it, I just don't need to do anything else with it right now. You'll probably know what to do with the ongoing part at the end. I'm just real tired right now, thank you again