Posted by: Jan
« on: Yesterday at 09:36:17 pm »What you are trying to achieve — loading and processing a complex web page directly in Magic RDS — is unfortunately not the right approach, for several fundamental reasons:
Recommended approach
The correct solution is to use a simple and dedicated data source, for example:
A plain text file (TXT)
A simple XML or JSON endpoint
A lightweight server-side script that extracts and provides only the required text
In other words, the Magic RDS can only retrieve a single file which should ideally contain a simple and consistent text data structure or the already prepared, clean text. The text in this form is always present on the web server or in associated database. Contact support to gain access to this data.
- Magic RDS is not a web browser
Modern web pages are designed to be rendered by full-featured web browsers (such as Google Chrome or Mozilla Firefox), which include sophisticated HTML parsers, JavaScript engines, CSS rendering, and dynamic content handling. Magic RDS does not implement this kind of environment and is not intended to replace a browser. There is currently no simpler or lightweight alternative that can reliably interpret such complex, script-dependent pages. - Periodic fetching creates significant overhead and stability risks
To keep the displayed text up to date, the system would need to repeatedly download and process the page. With complex pages, this implies:
High CPU usage (due to parsing and potential script handling),
increased data transfer,
risk of instability or crashes.
This is far beyond what is practical or safe for a system like Magic RDS, which is designed for efficiency and reliability. - No reliable way to extract the required text
Even if the page could be loaded, there is still the unresolved problem of identifying and extracting the specific text you want.
In modern web pages, the desired content is often generated dynamically via JavaScript, meaning it is not directly available in the raw HTML. Without a full browser engine, there is no reliable way to locate and isolate the correct data.
Recommended approach
The correct solution is to use a simple and dedicated data source, for example:
A plain text file (TXT)
A simple XML or JSON endpoint
A lightweight server-side script that extracts and provides only the required text
In other words, the Magic RDS can only retrieve a single file which should ideally contain a simple and consistent text data structure or the already prepared, clean text. The text in this form is always present on the web server or in associated database. Contact support to gain access to this data.