[Manal]

WPExtractor - WordPress Blog Post Extractor in JSON Format

---

WPExtractor is a python-based tool specifically made for Artificial Intelligence-based projects to make datasets. This helps to collect data from blogs which can be used to train bot in many useful ways.
Features

• Automatically extract all posts from a WordPress website within seconds.
  
  
• Saves the data in the JSON file in the directory for you.
  
  
• Easily understandable JSON format to make your life easier
  
  
• Responsive developers. Just make an issue, we'll fix it for you
  

---

Usage
Usage:

Code: (Select All)

python main.py -u https://csrockers.in

By default, it will fetch posts from the website. To fetch pages, use the following.

Code: (Select All)

python main.py -u https://fulltimehosting.net --pages

Credits
Manal Shaikh & Somil Gumber.

[tbelldesignco]

Ooh I am curious about this further. I am working on creating an app for my business and I could possibly use something like this to feed content in JSON structs to my app. Thank you! I will defiantly be looking into this tool further!

[tiwil]

Just looking at the script. I never thought that Wordpress have that /wp-json/ directory. I tested with my Wordpress website and it shows all of my pages and posts. Nice works, Manal! This one can help us to show the posts for a non-WP website.

Out of topic, but, how did you find the URL to that JSON?

[tbelldesignco]

I just want to confirm how amazing this tool is - I’ve currently have my app fetching content from my site and populating dynamic content into the app, I will have to get screenshots up when I’m back on my MacBook.

[LightDestory]

Nice projects, it is sad when an open source software became an abandonware. You did a very nice job picking up it and reviving it.
I did a very fast read of the code and you seems to be using wp-json stuff, it is a good choice but why don't use the sitemap?

Moreover you could replace the ifs inside the error handling with a swtich, it does the same thing but it is more "appropriate" for that situation.

[fitkoh]

This looks like an interesting tool. I'm not familiar with python, but I can see it's value. Unfortunately I wasn't able to get it working (remember, python newbie).

First, I installed python.

~$ sudo apt-get install python

no problems there. Then I try to run main.py as recommended in your readme.

~$ python main.py -u https://url.url
Traceback (most recent call last):
  File "main.py", line 5, in <module>
    import requests
ImportError: No module named requests

Okay so something is missing. I do some googling and find first I need to install pip.

~$ sudo apt-get install python3-pip
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3-pip is already the newest version (20.0.2-5ubuntu1.1).

that one looks good. So lets try to install request...

~$ sudo pip3 install requests
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (2.22.0)

So I have requests installed but for some reason my python installation isn't recognizing it. I'm thinking maybe the problem is due to multiple versions of python installed but I'm not sure which one to keep or if there's a safe way to remove one without breaking the other. Perhaps it'd be simpler to reinstall and start from scratch? (I'm working on a dev server so no worries about losing anything).

[Manal]

Try using

Code: (Select All)

python3 main.py -u https://url.url

The tool is based on python3, and I recommend you install requests using pip3, as you installed.

This looks like an interesting tool. I'm not familiar with python, but I can see it's value. Unfortunately I wasn't able to get it working (remember, python newbie).

First, I installed python.



no problems there. Then I try to run main.py as recommended in your readme.


Okay so something is missing. I do some googling and find first I need to install pip.

that one looks good. So lets try to install request...


So I have requests installed but for some reason my python installation isn't recognizing it. I'm thinking maybe the problem is due to multiple versions of python installed but I'm not sure which one to keep or if there's a safe way to remove one without breaking the other. Perhaps it'd be simpler to reinstall and start from scratch? (I'm working on a dev server so no worries about losing anything).

---

Nice projects, it is sad when an open source software became an abandonware. You did a very nice job picking up it and reviving it.
I did a very fast read of the code and you seems to be using wp-json stuff, it is a good choice but why don't use the sitemap?

Moreover you could replace the ifs inside the error handling with a swtich, it does the same thing but it is more "appropriate" for that situation.

Noted for next update.
The reason why I didn't use sitemap is sitemap may often missout on posts that are outside the website. This tool aims at everything that is not draft or password from the main content.
And I just started learning Python. This is just a pet project :3

[Sn1F3rt]

Great project @Manal! It works perfectly. Moreover, it's open-source, I took a look at the source code too.

I'm myself a Python Developer and glad to see you are starting with Python, you won't regret it.