On the site of today's chateau stood a second, younger fortress, which was probably built in the mid-15th century after the older fortress, which was located southwest of Kácov on the left bank of the Sázava River near Soušice, was abandoned. As Kácov was divided between several owners from the turn of the 14th and 15th centuries, the possibility that two fortresses existed side by side for a certain period of time cannot be ruled out.
The fortress on the site of the chateau is dated 1473, when three quarters of it belonged to Kuneš of Olbramovice. The builder chose a site in the southern part of Kácov on the edge of a rocky slope falling steeply to the Sázava River. This slope formed a natural fortification, which was supplemented on the northern side by an artificial fortification, which has now completely disappeared. The original form of the fortress cannot be reconstructed after several reconstructions.
In 1627, during the reign of Jan Oktavián Kinský, a serf uprising broke out on the Kákovský estate and the rebels plundered the fortress. It was not until 1630 that Benigna Katherine of Lobkowitz had it repaired and enlarged. A small one-storey Baroque chateau was built, with a stone ground floor and a wooden first floor. The chateau was further extended under the ownership of František Scheidler. Between 1727 and 1733, the Tuscan Princess Anna Maria Franziska had the ruined castle in Kács, which had been neglected by its previous indebted owners, rebuilt in the style of a Baroque northern Italian noble residence. The reconstruction completely changed the appearance of the former fortress area, removed the last remnants of the fortifications and only some of the foundation walls of the old building remain. The new castle was built on uneven terrain, so its southern facade facing the Sázava River was two-storeyed, while the northern facade, which included the main entrance, was single-storeyed. The castle was based on a rectangular plan, with the shorter western and eastern sides decorated with tower-like projections. These projections were as high as the main building and were therefore accentuated by Baroque onion-shaped towers. Inside, the axis of the castle building was a corridor running through both floors, which at its highest part surmounted the roof of the castle in the form of a rectangular half-moat, thus creating a silhouette typical of the castle. The corridor was the entrance to most of the rooms. The interiors were decorated mainly with stucco. The richest decoration was in the chateau chapel. The facade was decorated with murals representing various figures in the blind windows. The castle was separated from the square by a plain wall with a Baroque gate. The castle has been preserved in this form practically until today.
The chateau complex consists of three parts: the chateau itself, the parish church and the ash yard. A 150-metre-long connecting corridor between the chateau and the church, which is now inoperative and until recently was used for state archival purposes, completes the complex in a fundamental way. The whole historic complex was then further altered after 1920 by the construction of a falconry on the site of the granary. However, this change was preceded by the removal of the cemetery complex and its walls around the church on the basis of modern hygiene regulations. Only the chapel of St. Florian, the patron saint of firefighters, has remained from the cemetery enclosure, which is impossible to miss when driving through the town.