Creating a bootable ISO of OS X El Capitan is quite easy, if you have access to an OS X machine. You will need a copy of the "Install OS X El Capitan.app" which can be downloaded from the App Store. Once you have it, the application will be in your applications list.
Inside of /Applications/Install OS X El Capitan.app, there is an installation image (InstallESD.dmg). By itself, it is not bootable and cannot be used to install the system on a virtual machine or on an iMac or MacBook without some work. Were going to focus on getting the ISO for installing OS X on a virtual machine. Start off by mounting the InstallESD.dmg image using the tool hdiutil
$ hdiutil attach "/Applications/Install OS X El Capitan.app/Contents/SharedSupport/InstallESD.dmg" -noverify -nobrowse -mountpoint /Volumes/esd
The switches -noverify and -nobrowse make mounting faster and keep the finder application from automatically popping up with a new window. The -mountpoint switch is where we will get the files to copy over. Now it's time to create a placeholder image file for the ISO.
$ hdiutil create -o ElCapitan.cdr -size 7316m -layout SPUD -fs HFS+J
This operation can take a while. When it is complete, you should now have the file ElCapitan.cdr.dmg and it will need to be mounted to copy the files from the installation image we mounted.
$ hdiutil attach ElCapitan.cdr.dmg -noverify -nobrowse -mountpoint /Volumes/iso
Apple has a tool called asr (Apple Software Restore) which will be used to copy the contents from /Volumes/esd to /Volumes/iso
$ asr restore -source /Volumes/esd/BaseSystem.dmg -target /Volumes/iso -noprompt -noverify -erase
The asr tool will create a newly mounted drive called /Volumes/OS X Base System. This will be used to copy over some files and delete a symbolic link to the packages folder. To remove the invalid link that is not needed run this command:
$ rm /Volumes/OS\ X\ Base\ System/System/Installation/Packages
Instead of the invalid link, we need the actual directory which is elsewhere on the image. To copy over the entire directory:
$ cp -rp /Volumes/esd/Packages /Volumes/OS\ X\ Base\ System/System/Installation
Once that is complete, there are some additional dependencies needed for the installer.
$ cp -rp /Volumes/esd/BaseSystem.chunklist /Volumes/OS\ X\ Base\ System/
$ cp -rp /Volumes/esd/BaseSystem.dmg /Volumes/OS\ X\ Base\ System/
The other images can be unmounted, if everything worked properly the ElCapitan.cdr.dmg has everything we need.
$ hdiutil detach /Volumes/esd
$ hdiutil detach /Volumes/OS\ X\ Base\ System
For the last step, we need to convert the IS file into the UDTO format.
$ hdiutil convert ElCapitan.cdr.dmg -format UDTO -o ElCapitan.iso
This will output the ElCapitan.iso.cdr file, which can be renamed to ElCapitan.iso and be used in virtualization software like VMware ESXi or Oracle's VirtualBox. The method described above can also work with OS X Yosemite or Mavericks.
