Coffee Space


Listen:

Akregator RAM Usage

Preview Image

Preview Image

NOTE: This article is currently a work in progress. It may not have an interesting resolution, but you get to see the process in action.

Background

I used to use Thunderbird for my RSS feeds, but a while back it entirely corrupted my feed and I had to put in a lot of effort to recover my subscriptions. I managed to get back to an OPML file and simply decided in the end to move to a different feed reader.

After looking around, I found akregator seeming to be a simple, old and well supported feed reader. From what I can tell it compiles down to a binary, rather than being one of these interpreted language programs that one day decides it doesn't want to work, because a Node package that does some simple task is suddenly deleted.

Akregator has some problems, like not being able to switch off the tabs system (I only want to download content or view it in a browser), supporting Javascript (my feed reader shouldn't have a Javascript engine, that's what my browser is for) and when it crashes it forgets what you have already read (which seems pretty dumb).

That said, it generally has been more reliably than Thunderbird so far, which probably is a pretty low bar to pass - given it corrupted my feeds and silently stopped updating them.

Problem

The other day my laptop running Ubuntu started freezing up and completely locked up. All software updates are disabled (including snap), which was the previous cause of instability. After running for weeks, I was surprised that it suddenly became so unstable.

I of course was forced to reboot my machine manually and was quite cautious thereafter, checking back at the System Monitor regularly. To my surprise. I see akgregator of all things consuming over 1GB of RAM! It was giving Firefox a run for its money, and my version of Firefox has over 1000 tabs open! What the hell!

Debugging (2021-08-04)

First things first, I went to DuckDuckGo and searched for similar bugs - to which I found none. This was kind of annoying, so I sat on the problem a little longer, played with the settings and the RAM usage continued to increase.

I then went to IRC on the #kde-bugs channel on the Libera server. There somebody recommends to lodge a bug report over on KDE. I check for existing bugs there to do a cursory glance to try and prevent replication of efforts. After scrolling through I find nothing.

After finding the relevant version information, I registered and submitted a report. Some details from that report:

SUMMARY

I am currently observing ~250MB of RAM usage increase per day. This lead to a
system crash.

Currently Akregator is consuming 1.7GB of RAM, 4.9GB of virtual memory.
Compare this to Firefox with hundreds of active tabs, consuming 1.4GB of RAM
and 8.9GB of virtual memory.

STEPS TO REPRODUCE
1. Large number of RSS feeds (OPML export is ~300kB and ~700 lines)
2. Set checking time to 30 minute intervals
3. Wait for 24 hours

OBSERVED RESULT

A ~250MB RAM increase over 24 hours. Over several weeks this will lead to a
system lock-up/crash (in this case).

EXPECTED RESULT

A little to no RAM increase over 24 hours.

SOFTWARE/OS VERSIONS
Windows: -
macOS: -
Linux/KDE Plasma: Ubuntu 21.04
(available in About System)
KDE Plasma Version: -
KDE Frameworks Version: 5.80.0
Qt Version: 5.15.2

ADDITIONAL INFORMATION

Then I thought to myself, perhaps there is more I can do.

After looking online at several solution for monitoring RAM usage for a given process, I settled on and adjusted an answer from StackOverflow:

delay=300
pid="$1"

top -b -d $delay -p $pid | awk -v OFS="," '$1=="top"{ time=$3 }
$1+0>0 { print time,$1,$NF,$9,$10; fflush() }'

After running for about 9 hours, we see this:

RAM usage study

RAM usage study

As you can see, RAM usage is increasing over time as I suspected. It appears to be increasing approximatly 20MB per hour and is mostly linear. There appear to be two lines, perhaps the top one is when it is actively processing new incoming RSS feeds.

Other than this, it's still too early to draw conclusions. I now continue to wait for 24 hours and see if somebody becomes interested in my ticket.

Debugging (2021-08-05)

As I suspected, things didn't get better inside the 24 hours observation time:

24H RAM usage study

24H RAM usage study

Next steps:

I have begun running valgrind using the advice from StackOverflow:

valgrind --leak-check=full -v --track-origins=yes --log-file=log.txt valgrind

I did find that --track-origins=yes was exceptionally slow, so I removed this to get something like a usable experience. I will run for a few hours before closing to ensure I definitely get some data.

After testing - is there a leak? Yes. From the log we see:

==595152== LEAK SUMMARY:
==595152==    definitely lost: 298,872 bytes in 357 blocks
==595152==    indirectly lost: 2,469,366 bytes in 10,308 blocks
==595152==      possibly lost: 232,592 bytes in 476 blocks
==595152==    still reachable: 10,728,421 bytes in 104,272 blocks
==595152==                       of which reachable via heuristic:
==595152==                         newarray           : 936 bytes in 5 blocks
==595152==                         multipleinheritance: 29,552 bytes in 21 blocks
==595152==         suppressed: 0 bytes in 0 blocks
==595152== Reachable blocks (those to which a pointer was found) are not shown.
==595152== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==595152==
==595152== Use --track-origins=yes to see where uninitialised values come from
==595152== ERROR SUMMARY: 14687 errors from 1145 contexts (suppressed: 0 from 0)

From what I can tell (with my very limited knowledge), we see to be leaking memory mostly from libQt and more specifically libQt5WebEngineCore:

==595152== 836,845 (186,208 direct, 650,637 indirect) bytes in 92 blocks are definitely lost in loss record 25,600 of 25,601
==595152==    at 0x4842FB3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==595152==    by 0x744FCC9: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
==595152==    by 0x77B2AB2: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
==595152==    by 0x9E17A8A: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
==595152==    by 0x9E184AE: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
==595152==    by 0x9E1C912: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
==595152==    by 0x9E1F664: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
==595152==    by 0x9E14D44: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
==595152==    by 0x9E1505D: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
==595152==    by 0x9E32CE0: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
==595152==    by 0x97700DA: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
==595152==    by 0x9783A93: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)

To be sure, I would probably need to compile from source. If somebody does eventually answer the ticket, they will likely ask me to use the most up-to-date version and confirm the problem still persists there too. For now though I will wait for somebody to reply and keep logging RAM increase over time.

Debugging (2021-08-06)

After running for a while longer, the trend appears to still hold true, even from a recently started process:

24H RAM usage study

24H RAM usage study

I thought some more about building from source, and after a little searching, found Akregator on GitHub. I saw there a comment about a dedicated channel on IRC, but haven't been able to get hold of anybody there yet:

IRC:  irc://irc.libera.chat/#akregator

I asked about build dependencies on IRC, but got no comment there. I decided I would just give it a go and see what happens.

First of all, we need to clone and try an initial build:

cd $WORKING DIRECTORY # Got to where we want to clone it
git clone git@github.com:KDE/akregator.git
cd akregator/         # Go to where we clone it
gitk --all            # Check everything looks good
git checkout v20.12.3 # Checkout the version I want to build
cmake .               # Attempt to generate the Makefile

It gives it a go, but fails:

-- The C compiler identification is GNU 10.3.0
-- The CXX compiler identification is GNU 10.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:8 (find_package):
  Could not find a package configuration file provided by "ECM" (requested
  version 5.75.0) with any of the following names:

    ECMConfig.cmake
    ecm-config.cmake

  Add the installation prefix of "ECM" to CMAKE_PREFIX_PATH or set "ECM_DIR"
  to a directory containing one of the above files.  If "ECM" provides a
  separate development package or SDK, be sure it has been installed.


-- Configuring incomplete, errors occurred!
See also "./CMakeFiles/CMakeOutput.log".

So here we can see it failed to build based on ECM, so we need to find out what it is complaining about and likely install something to meet that dependency. This is the list of things I ended up needed to download via sudo apt-get install <PACKAGE> (your milleage may vary):

Finally I was able to run cmake . and get a good message (the list below of REQUIRED packages may give you a better list for installation):

Installing in /usr/local. Run ./akregator/prefix.sh to set the environment for akregator.
-- Could NOT find KUserFeedback (missing: KUserFeedback_DIR)
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- The following OPTIONAL packages have been found:

 * QGpgme

-- The following REQUIRED packages have been found:

 * ECM (required version >= 5.75.0)
 * Qt5Qml (required version >= 5.15.2)
 * Qt5QmlModels (required version >= 5.15.2)
 * Qt5Quick (required version >= 5.15.2)
 * Qt5WebChannel (required version >= 5.15.2)
 * Qt5Positioning (required version >= 5.15.2)
 * Qt5WebEngineCore (required version >= 5.15.2)
 * Qt5WebEngineWidgets
 * Qt5PrintSupport
 * Qt5 (required version >= 5.13.0)
 * Grantlee5 (required version >= 5.2)
 * KF5Crash (required version >= 5.75.0)
 * KF5DocTools (required version >= 5.75.0)
 * KF5KCMUtils (required version >= 5.75.0)
 * KF5NotifyConfig (required version >= 5.75.0)
 * KF5SyntaxHighlighting (required version >= 5.80.0)
 * KF5TextEditor (required version >= 5.75.0)
 * KF5Notifications (required version >= 5.75.0)
 * KF5Syndication (required version >= 5.75.0)
 * KF5Parts (required version >= 5.75.0)
 * KF5KontactInterface (required version >= 5.16.3)
 * KF5Libkdepim (required version >= 5.16.3)
 * KF5MessageCore (required version >= 5.16.3)
 * KF5MimeTreeParser (required version >= 5.16.3)
 * KF5MessageViewer (required version >= 5.16.3)
 * KF5Sonnet (required version >= 5.80.0)
 * KF5PimTextEdit (required version >= 5.16.3)
 * Qt5WebEngine (required version >= 5.13.0)
 * KF5WebEngineViewer (required version >= 5.16.3)
 * KF5AkonadiMime
 * KF5ItemModels (required version >= 5.75.0)
 * Qt5Gui (required version >= 5.13.0)
 * KF5Service (required version >= 5.80.0)
 * KF5Completion (required version >= 5.80.0)
 * KF5ItemViews (required version >= 5.80.0)
 * KF5JobWidgets (required version >= 5.80.0)
 * KF5Solid (required version >= 5.80.0)
 * KF5Auth (required version >= 5.80.0)
 * KF5WidgetsAddons (required version >= 5.80.0)
 * KF5ConfigWidgets (required version >= 5.80.0)
 * KF5XmlGui (required version >= 5.80.0)
 * KF5WindowSystem (required version >= 5.80.0)
 * Qt5Network (required version >= 5.14.0)
 * Qt5Test (required version >= 5.13.0)
 * Gettext
 * KF5Contacts (required version >= 5.75.0)
 * KF5GrantleeTheme (required version >= 5.16.3)
 * Qt5Widgets (required version >= 5.13.0)
 * KF5AkonadiContact (required version >= 5.16.3)
 * KF5CoreAddons (required version >= 5.75.0)
 * Qt5Core (required version >= 5.14.0)
 * KF5Codecs (required version >= 5.75.0)
 * KF5Mime (required version >= 5.16.3)
 * KF5IMAP (required version >= 5.16.3)
 * KF5PimCommonAkonadi (required version >= 5.16.3)

-- The following OPTIONAL packages have not been found:

 * KUserFeedback (required version >= 1.0.0), User Feedback lib
   Allow to send Telemetry Information (optional). It can be disable in apps.

-- Configuring done
-- Generating done
-- Build files have been written to: ./akregator

And now we can see the built binaries:

$ ls bin/
akregator                       akregatorstorageexporter
akregator_config_advanced.so    downloadfeediconjobtest
akregator_config_appearance.so  kontact_akregatorplugin.so
akregator_config_archive.so     libakregatorinterfaces.so
akregator_config_browser.so     libakregatorinterfaces.so.5
akregator_config_general.so     libakregatorinterfaces.so.5.16.3
akregator_config_plugins.so     libakregatorprivate.so
akregator_mk4storage_plugin.so  libakregatorprivate.so.5
akregatorpart.so                libakregatorprivate.so.5.16.3

Next we run the program and make sure the memory leak still exists in the version we just built from source:

./bin/akregator

This will take a little while, so I will follow up in a further update. The only other thing to do was update the ticket regarding my current actions and plans.

Debugging (2021-08-07)

After nearly 24 hours, we observe our RAM study on the built-from-source version:

24H RAM usage study

24H RAM usage study

Some interesting observations:

./bin/akregator: symbol lookup error: /usr/lib/x86_64-linux-gnu/qt5/plugins/akregator_config_userfeedback.so: undefined symbol: _ZN9Akregator19UserFeedBackManager4selfEv

The built version suggests the following versions (supposedly the same as before):

Not entirely sure where the different is coming from here... Time to update the ticket!

Debugging (2021-08-08)

I recieved a comment on the ticket that suggests I should remove the local copy of Akregator first:

You rebuild akregator but it seems that it uses akregator_config_userfeedback.so from your system not you install directory. If you rebuild remove your akregator from your system first.

I run sudo apt-get remove akregator and sure enough, it removes userfeedback:

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  kuserfeedback-doc libkf5messageviewer-plugins libkuserfeedbackcore1
  libkuserfeedbackwidgets1 qml-module-org-kde-userfeedback
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED
  akregator
0 to upgrade, 0 to newly install, 1 to remove and 25 not to upgrade.
After this operation, 6,495 kB disk space will be freed.

After running cmake ., I could see I need to install kuserfeedback-dev with:

sudo apt-get install kuserfeedback-dev

After this I rebuild Akregator to ensure it's built against local sources:

make clean
make
sudo make install # Install metakit

Now I am re-running the RAM study to see if the issue replicates. Lastly, I update the ticket and we see how things go.

Debugging (2021-08-09)

After running Akregator for about 24 hours, we finally see we are able to replicate the issue when building from source (and in debug mode):

24H RAM usage study

24H RAM usage study

Next we want to run Akreagtor under valgrind so that we can get more useful debug information:

valgrind --leak-check=full -v --track-origins=yes --log-file=akregator-vg-2021-08-09.txt ./bin/akregator

Again, running with --track-origins=yes is insanely slow, to the point where I'm not sure the program is usbale and able to replicate the original issue.

Lastly, we update the ticket.

The next steps will be:

Debugging (2021-08-10)

The ticket has been updated with the valgrind output.

The next few days will likely be busy, hence I will need to move onto other projects. I will give the person responding on the ticket a chance to check things over and make recommendations - as I'm essentially just stabbing in the dark at this point.

Debugging (2021-08-13)

No further updates at this time - except I may have a branch of Akregator to checkout with a different archive back-end that should reduce RAM usage. Apparently, the default behaviour of Akregator is to try and store everything in RAM in order to keep some speed in the GUI.

Given how fast disks are (even old IDE drives), they should be more than fast enough to fetch some feed titles in a fast (undetectable) time. A design for any program that means it slowly uses more and more RAM over time - seems awful and broken. I'm beginning to believe that I might be better off rolling my own feed agreagtor. I don't have the time for that right now, but I really suspect it really is the right way forwards.

Debugging (2021-09-02)

Again, no further updates to report. Still not heard anything else from the developer. As time goes on, the idea of rolling my own feed agregator becomes more attractive. I suspect that the software would require a large overhaul as this is a design flaw - currently it tries to store everything in RAM.

I'm going to have to neglect this from my mind as I have more important things to work on. If I come back to it randomly in the future and there is still no update, I will consider it a lost cause.

Conclusion

Stay tuned for more!