Magnolia 5.6 reached end of life on June 25, 2020. This branch is no longer supported, see End-of-life policy.
...
Magnolia 5.5.4 6 ee-bundle may require you to allocate more memory the Java Virtual Machine (JVM). If you see a java.lang.OutOfMemoryError
in the startup log or the system stops responding during installation, increase the Java heap size. The default maximum heap size is 512M. Try a higher amount such as 1024M. We are working on uncovering the root cause for the increased memory need.
...