LaTeX is an open source markup language with a typesetting engine. It's been around since the 70s and generally makes awesome documents.
LaTeX is designed to work stand-alone, or integrate with other languages. It's particularly good with R.
It works in the way rmarkdown does with knitr, but allows for more sophisticated document styling.
Once you have a LaTeX installation, you can write LaTeX in Rstudio.
In Rstudio, select the File type “R Sweave” which saves as a .Rnw file.
LaTeX is a very deep language, so no attempt is made to teach you here.
One of the easiest ways of getting started with LaTeX (generally) is to pick one of the examples / templates on overleaf.com and play with it.
For using R and LaTeX, you can use the minimal examples on yihui.name/knitr to get started.
See the LaTeX wikibook for lots of info about LaTeX.